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I. INTRODUCTION 

A country must have a strong and vibrant research and development capacity if it is to thrive in 
the rapidly emerging post-industrial world. To achieve this end, most of the industrialized nations 
have decided that government must play a prominent role in assembling and sustaining a sound 
scientific and engineering infrastructure. Most modem governments provide financial support for 
other professional endeavors, such as the fine arts, though usually on not as grand a scale. Although 
many procedural vehicles exist for making the difficult scientific and artistic judgments that 
necessarily arise in deciding which of a large pool of applicants will receive limited governmental 
resources, the United States has depended to a very large degree upon "peer review" grant systems in 
which the governmental decisionmaker assembles a group of the grant applicants' peers for advice. 
Under this peer review model, the government does not attempt to persuade researchers to undertake 
particular research or artists to create particular kinds of art. Instead, a granting agency allocates 
sums of money to entire fields and allows the researchers, artists, or performers to develop creative 
proposals for projects.' A group of "peers" with expertise in the relevant area then evaluates and 
ranks proposals, leaving the ultimate funding decisions up to the governmental officials in charge of 
the relevant programs. 

The peer review process has proved remarkably durable in the 30-40 years during which federal 
agencies have employed it. It is, however, far from perfect. This Report will focus on one 
especially worrisome criticism of peer review systems for awarding discretionary grants~that they are 
capable of both ad hoc and systematic bias for and against individuals, groups, and innovative new 
ideas." After taking some measure of the validity of this criticism, the Report will compare the peer 
review model to the primary competing model for awarding discretionary grants— the "strong 
manager" model. Concluding that on balance the peer review model is most appropriate for 
awarding discretionary grants in the arts and sciences, the Report will suggest some procedural 
reforms that should help reduce the potential for bias at a relatively little cost. 

A. The Nature of the Peer Review Process 

A "peer review" system of allocating governmental resources relies primarily upon the informed 
recommendations of experts in the relevant field of inquiry.' First employed more than three hundred 
years ago for evaluating the scientific merit of journal articles, it was adopted by the National 
Advisory Cancer Council and the Office of NavaJ Research in the 1930s and 1940s for assessing 
applications for scientific grants. In the ensuing years the peer review model has evolved into a 
highly regularized vehicle for awarding research grants that "legitimates the flow of resources and the 
establishment of priorities" in the scientific granting agencies. -• By the mid-1980s the Board of 



'The above dcBcription is somewhat idealized to the extent that it suggests that the funding agencies do not attempt to steer research. 
Obviously, budgetary decisions about which programs get what funds send messages to potential researchers about where they should 
direct their efforts. Sometimes the signals can be more direct as, for example, when NSF refused to continue funding the one of five 
supercomputer centers that used a particular brand of computer. See Ellis Booker, NSF Cuu Supercomputer Grant, Compute rworld, 
October 30, 1989, at 127. 

^The Report will not examine the closely related question of whether the funding programs themselves are biased toward or against 
funding research aimed at benefiting particular groups. The report will, however, examine claims that the system is biased for or against 
particular researchers. For example, one of the most frequent criticisms of peer review systems is that they result in 'old boy' networlu. 

'Chubin and Hackett defme peer review as: 'An organized method for evaluating scientific work which is used by scientists to certify 
the correctness of procedures, establish the plausibility of results, and allocate scarce resources (such as journal space, research funds, 
recognition, and special honor).* D. Chubin &. E. Hackett, Peerless Science: Peer Review and U.S. Science Policy 20 (Slate University of 
New York Press, 1990). fhereinafler cited as Chubin & Hackett} A House subcomminee report on peer review in NSF defmes 'peer 
review system* as 'any method of evaluating a specialized creation - such as a proposal to perform scientific research - which involves 
having a group of people knowledgeable in the area of specialization evaluate the creation* National Science Foundation Peer Review: 
Report of the Subcommittee on Science, Research and Technology of the House Committee on Science and Technology, 94ih Cong., 2d 
Sess (1976), at 13 [hereinafter cited as 1976 House Hearing Reports]. 

*D. Chubin &. S. JasanofT, Peer Review and Public Policy, 10 Science, Technology & Human Values 3 (1985). See also Chubin &. 
Hackett, supra note 3. 
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Directors of the American Association for the Advancement of Science could conclude that "the 
scientific community has accepted primary responsibility for defining research needs and 
opportunities and providing insurance that public funds are allocated on a priority basis through peer 
review."^ It is nevertheless true that "[sjcience flourishes all over the world without the use of 
individual proposal peer review."* 

Peer review in the arts is of even more recent vintage. Until the mid-1960s, private arts patrons 
rarely invoked formal peer review as a vehicle for making funding decisions.' With the creation of 
the National Endowments for the Arts and Humanities, however, the federal government became the 
nation's most prominent single fine arts patron, and both institutions borrowed the peer review model 
from the scientific funding agencies. As with the scientific agencies, peer review has helped 
legitimate NEA funding decisions among the practitioners of the arts, though not necessarily among 
members of the general public' 

As it has evolved in the United States, peer review is intended to ensure that public funds are 
awarded to the most meritorious scientific and artistic projects in a way that renders the system 
accountable to the public and its elected representatives. And herein lies the potential for 
considerable tension. As long-time observers Chubin and Hackett note: 

[P]eer review is expected to operate according to values of fairness and expediency 
yet its product is to be trustworthy, high-quality, innovative knowledge. There is no 
assurance that the process will yield the product; to the contrary, the process may 
interfere with efforts to secure the product.' 

B. Bias in Decisionmaking 

Perhaps the most important goal of the peer review process is objectivity. Governmental 
allocation of monies to fund projects in the arts and sciences is built upon the assumption that 
objective criteria for excellence can be articulated and applied in a way that is capable of identifying 
meritorious proposals and of selecting the best from among those. To the extent that bias infects the 
decisionmaking process, it loses its objectivity and, consequently, its legitimacy. 

The word "bias" takes on different meanings in different contexts. For example, in 
Administrative Law the courts consider a decisionmaker biased when he or she has a "irrevocably 
closed" mind with respect to the facts.'" In the context of peer review, however, the concept comes 
closer to the dictionary definition of "a highly personal and unreasoned distortion of judgment."" A 
biased decisionmaker does not decide questions on their merits, but rather allows irrelevant personal 
considerations to intrude systematically into the decisionmaking process. Decisionmaking bias in the 
award of discretionary grants can result from favoritism, animus, or conflict of interest. The first 
two stem from the identity of the potential grantee; the latter has more to do with the characteristics 
and position of the decisionmaker. Bias can also reflect "tunnel vision," or the systematic refusal to 
give sufficient weight to particular criteria that are supposed to be relevant to the decision. Finally, 
ex parte lobbying and political pressure can cause an otherwise objective process to become biased 
for or against particular persons or approaches. 



^American Association for the Advancement of Science Board Statement on Politics and Science, 223 Science 27 (Jan. 1984). 

'R. Roy, Funding Science: The Real DefecU of Peer Review and an Alternative To It, 10 Science Technology and Human Values 73, 
74 (Summer 1985). 

^lan M. Kriegsman, "The Dance Dilemma: Taking Slock at a Meeting of Moguls,' Washington Post, July 17. 1983, LI (corporations 
'don't have a peer review panel system for determining what dance to support-it's unique to NEA*) (quoting Mr. Richard LeBlond). 

'Joseph McLellan, NEA: The First Twenty Years, Washington Post. Sept. 16. 1985, Dl (peer review *6eem[s] to have generated 
considerable respect for the NEA among American artists, who feel they are being evaluated not by bureaucrats but by fellow artisu.*) 

'Chubin & Hackett, supra note 3, at 3. 

"^Federal Trade Conun'n v. Cement Institute, 333 U.S. 683, 701 (1948). See 3 Kenneth C. Davis. Administrative Law Treatise 

19:2, 19:4; 19:7 acd. 1980). 

"Webster's Ninth New Collegiate Dictionary (1988). 
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1. Aaimus 

Animus is the ugliest aspect of bias. A governmental assistance program is affected by animus if 
an applicant's prospects of receiving a grant is adversely affected by some characteristic of that 
person unrelated to the articulated criteria. Another word for animus in this context is 
"blackballing." The animus may result from any one of a number of causes, ranging from personal 
dislike to philosophical differences to racial bias. Animus can stem from personal dislike of an 
individual, or it can display a general lack of regard for "mavericks" who challenge conventional 
norms. It may be manifested in many ways, ranging from introducing factually inaccurate 
information into the decisionmaking process to ranking the applicant at the bottom of the list. 

2. Tunnel Vision 

Objectivity is also lost when the process becomes afflicted with "tunnel vision," a malady that 
obstructs the ability of highly trained professionals to view proposals from different perspectives. 
The problem is not so much invidious discrimination as it is the inability or unwillingness to see the 
relevance of entire categories of proposals. The tendency becomes a disadvantage when it operates to 
exclude otherwise qualified proposals that come within the statutorily or administratively drawn 
bounds of the program. A process that screens out irrelevant proposals is not biased; a process that 
excludes relevant proposals because the decisionmakers have an unduly narrow professional view of 
the domain of relevance may be biased. 

3. Favoritism 

Favoritism is probably much more prevalent than animus in the peer review system. A decision 
is affected by favoritism when one or more of the applicants in the applicant pool obtains favorable 
treatment for reasons that are not relevant to the statutory or administrative criteria. The favoritism 
may manifest itself in many ways, ranging from unmerited high ranking to explicit singling out of an 
individual for the grant award. Nepotism is one form of favoritism; cronyism is another. The 
former is almost nonexistent in science, but the latter may play a role in peer review granting 
agencies. Favoritism is not necessarily limited to peer reviewers. The agency staff can also play 
favorites by selecting reviewers who are inclined to fund some kinds of proposals and not others.'^ 

4. Conflict of Interest 

A conflict of interest exists when a decisionmaker's judgment is clouded by his or her own 
personal stake in the outcome of the decisionmaking process. The clearest conflict of interest exists 
when the decisionmaker will obtain financial gain or suffer monetary loss if the decision goes one 
way rather than the other. For example, if one of the peer reviewers will share in the proceeds of the 
grant under review, the reviewer's judgment could be clouded by the prospect of financial gain. A 
reviewer also faces a conflict of interest if, through a consultantship or other arrangement with a 
company, the reviewer can convey financially valuable information to the for-profit enterprise with 
which he or she is affiliated. This familiar form of conflict of interest will be referred to here as 
"financial conflict of interest." 

A less dramatic, though perhaps more prevalent, form of conflict of interest might be 
characterized as "research conflict of interest." An active researcher is always on the lookout for 
new ideas that have not been tried, previously undiscovered approaches that have succeeded, and 
novel approaches that have failed but have future potential. Even if a reviewer does not directly 
appropriate an idea from a grant application, much can be learned from the review process that is not 
available in the published literature. Knowledge gained during the review process may steer a 
reviewer away from a line of inquiry that has proved fruitless for other researchers (and therefore not 



'^oy, Funding Science, supra note 6, at 73-81. (Peer review 'is susceptible lo manipulation by nrtanagers who operate under the 
emotional cover of the 'peer review' rubric so that individual proposals may be favored or disfavored simply by appropriate selection of 
the reviewers.") 
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published in the literature), or it may trigger an idea that, given sufficient time, would have become 
apparent to the applicant. 

Conversely, a reviewer who has established himself in one field of inquiry may tend to 
discourage the development of alternative lines of inquiry that may pose a threat to the reviewer's 
prominence. By simply downgrading a competitor's proposal, a reviewer can simultaneously reduce 
the competitor's probability for success and enhancing the probability that the reviewer's research 
will continue to receive support. This can be especially troubling when the scientific field at issue is 
in the process of a "paradigm shift" in which younger scientists question the power of the old 
paradigm to explain new phenomena, seek out different formulations of the critical questions, and 
advance alternative theories to explain observations." If the granting agency review panels are filled 
with devotees of the old paradigm, the development of such new paradigms might be impeded. 

5. Lobbying and Political Pressure 

In legislative decisionmaking bodies the most common vehicle for persuading the 
decisionmaking entity to send resources in one direction or another is the lobbyist. In the lobbying 
paradigm, the potential recipient of the funds or a paid representative of the potential recipient 
attempts to make as many contacts as possible with persons in a position to determine the outcome of 
the process and to persuade those persons of the wisdom of appropriating resources to that 
supplicant. There is a long American tradition of bringing political pressure to bear on 
decisionmakers in the bureaucracies. In the context of discretionary grants, prospective grantees can 
attempt to persuade influential higher-ups in the Administration or important congresspersons to "go 
to bat" for their proposals. 

Although the lobbyist is a well respected participant in the legislative appropriations process, the 
legislative model does not necessarily lead to the most objective results. Rather than awarding grants 
on the merits of the applications, the process appears to favor insiders and influence peddlers. A 
decisionmaking process in which the results depend upon political access and subtle promises and 
threats is not likely to value objectivity very highly. It could yield the impression that the results are 
biased in favor of the politically well-connected, and in the extreme could give rise to charges of 
partisan favoritism. 

C. Organization of this Report 

The next four Parts of this report examine several discretionary grants programs in four granting 
agencies: the National Institutes of Health (NIH); the National Science Foundation (NSF); the 
Environmental Proteaion Agency (EPA); and the National Endowment for the Arts (NEA). The 
facts and analysis contained therein are drawn from an extensive literature on peer review in NIH and 
NSF and a growing literature on the NEA grants process. More importantly, the description draws 
upon extensive interviews conducted with agency officials and in some cases successful and rejected 
applicants. All interviewees were assured that their responses would be kept confidential. Several 
graciously consented to allow their interviews to be cited for attribution; most did not want their 
names associated with the report in any way. In all cases the interviewees' wishes were honored. 
Candor was far more important to the author than the ability to attribute a factual statement to a 
particular source. 

Although all of these agencies rely heavily upon the principal of peer review in awarding 
discretionary grants, there are numerous variations across the agencies in how they manage the peer 
review process. None of the programs has completely eliminated the potential for bias, but some 
have made great strides in that direction. Each can learn lessons from the others, and the many 
granting agencies not covered here can learn a great deal from their combined experience. 



'^Scc T. S. Kuhn, The Sirocture of Scientific Revolutions (1962) 
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After describing the peer review process in four real-world contexts, Part 6 of the report will 
analyze in detail the potential for bias in peer review systems and examine the damage that the 
potential for bias can do to the integrity of the process. Finally, Part 7 will identify and analyze 
several possible solutions to the bias problem in peer review system, ranging from the radical 
suggestion that peer review be abandoned in some contexts to relatively minor "patch and repair" 
solutions to problems that have arisen in some programs. The report concludes that peer review is a 
very effective tool for ensuring objectivity in awarding grants in professional contexts. For the most 
part, favoritism, animus, tunnel vision, conflict of interest, lobbying and political pressures do not 
play large roles in existing peer review systems. While this is a comforting conclusion, some 
improvements can be made and some changes are in order. 
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II. PEER REVIEW IN THE NATIONAL INSTFTUTES OF HEALTH 

The National Institutes of Health (NIH) is a collection of research institutions in the Public 
Health Service of the Department of Health and Human Services.'" One of the premier research 
institutions in the World, it provides almost $7 billion per year to support more than 25,000 separate 
research awards in health and environmental sciences. In a very real sense, NIH is the father of the 
biotechnology industry in the United States.'^ NIH is compost of thirteen separate institutes, each 
of which directs its attention to a particular area of health-related research.'* Much of the research 
that the NIH sponsors is conducted by federal employees at the flagship campus in Bethesda, 
Maryland and at several other research centers around the United States. In addition to conducting 
its own research, NIH sponsors billions of dollars worth of research annually at Universities and 
other public and private research institutions.'^ Virtually all of these outside grants, contracts and 
interagency agreements are awarded through an intricate process that relies almost exclusively upon 
peer review for evaluating the quality of grant proposals. 

A. Discretionary Grant Programs 

Since NIH has in recent years suffered from chronic underfunding, many more worthy 
applications are submitted than can be funded.'* Therefore, the goal of the discretionary grant 
programs in NIH is to select the best applications from among a large number of very good 
applications. The competition among exceedingly qualified researchers for a shrinking pool of 
resources may itself threaten the integrity of the peer review process. When the stakes are so high, 
the temptations to cut comers and apply inappropriate criteria may be higher and the need to shield 
the process from bias correspondingly greater." Although all of the Institutes follow the same model 
for awarding discretionary grants, the following description of the NIH process will attempt to 
provide a feel for the variety of the grants that NIH funds by focusing particularly upon four very 
diverse grant programs: (1) the Program for Cancer Cause and Prevention Research of the Division 
of Cancer Etiology of the National Cancer Institute; (2) the Program for Biological Response to 
Environmental Health Hazards of the National Institute of Environmental Health Sciences; (3) the 
General Clinical Research Centers Program in the Division of Research Resources of NIH; and (4) 
the Biomedical Research Technology Program in the same Division of NIH. 

1. The National Cancer Institute 

The Program for Cancer Cause and Prevention Research, located in the Division of Cancer 
Etiology of the National Cancer Institute, awards grants, contracts and cooperative agreements to 
universities, colleges, hospitals, public agencies, nonprofit research institutions and for-profit 
organizations for projects designal to identify factors that cause human cancer and to develop 
mechanisms for preventing cancer. Research programs include studies in epidemiology, chemical 
and physical carcinogenesis, biological carcinogenesis, nutrition, immunology, and various other 
field studies and statistical research. The grants and cooperative agreements may be used for 
personnel, consultant costs, equipment, supplies, travel, patient costs, animals, alterations and 



'*The National Institutes of Health are established by the Health Research Extension Act of 1985. Pub. L. 99-158, 99 Sut. 820 
(1985). 42U.S.C. §201 et »eq. 

'^Thomas E. Malone, then I>eputy Director of NIH, said of NIH's role in the biouchnology industry: 'We fathered the industry. We 
are part and parcel of it." Nell Henderson and Michael Schrage, Govemnwnt R&D Funding Spawns a New Industry, Washington Post. 
December 16. 1984, Al col.l. 

'^ee National InsUtutes of Health, NIH Dau Book - 1989. 

'^NIH has only fiinded research in for-profit itutions since 1981, and applications from private companies still constitutes only a 
miniscule proportion of the total NIH budget. Se*. .nderson and Schrage, supra note 15. at Al col. 1. 

'*Sce Kolau, Beginning Scientists Face a Research Fund Drought, New York Times. June 5. 1990. CI , col. 2. 

"See Rosenberg, Medical Research b In Ruins, New York Times, Sept. 2, 1990 §4, p. 13. col. I (The chronic shortage of funding at 
NIH 'results in such a small fraction of meritorious grants being awarded that it jeopardizes the very objectivity and integrity of the peer 
review system for evaluating research— a system that depends on a vigorous and varied research community"). 
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renovation, miscellaneous items and indirect costs. ^ The Program allocates about $250,000,000 per 
year to cancer researchers in 300-400 awards that range from $2500 to more than $6,000,000 per 
grant. ^' About 25 percent of all applications are funded. - 

The Program also awards Small Business Innovation Research (SBIR) Grants to small 
businesses. The objectives of the SBIR awards are: "to stimulate technological innovations; to use 
small businesses to meet Federal research and development needs; to increase profit sector 
commercialization of innovations derived from Federal research and development; and to foster and 
encourage participation by minority and disadvantaged persons in technological innovation."^ 
Approximately $800,000 per year is available for a small number of SBIR grants and contracts. 

2. The National Institute of Environmental Health Sciences 

The National Institute of Environmental Health Sciences (NIEHS), located in Research Triangle 
Park, North Carolina, is the federal agency with the primary responsibility for conducting and 
supporting research on the effects of chemical, physical and biological environmental agents on 
human health. NIEHS attempts to prevent disease through programs that "provide the scientific 
information base, advanced methodology, and trained manpower to understand and prevent adverse 
health effects."^ Although about one-half of its work is done by its own staff,^ NIEHS supports 
more than 500 research and training grants that are designed to: "(i)dentify and characterize 
potentially harmftil environmental agents; fljearn the mechanism of action in biological systems after 
exposure to hazardous agents; [s]how what effects these agents cause, by themselves and in 
combination with other environmental factors; [l]earn what diseases are caused or aggravated by 
environmental factors; and to determine [t]he extent of exposure of various population groups to 
these agents."^ The grants are usually substantial and have an average duration of about four years. ^ 

The objectives of the Program are to identify and measure the biological, chemical and physical 
factors that are hazardous in the human environment as an essential first step in establishing the 
relationship between different levels of exposure to these factors and possible impacts on public 
health. The ultimate goal is to obtain a better understanding of the dose-response relationships 
between toxic substances and human exposures so as to improve public health protection against toxic 
environmental agents. In addition the Program aims at developing sensitive techniques for analyzing 
human exposures to environmental agents and for determining the rates at which such agents enter 
the body, how they are stored, metabolized and excreted. The Program also seeks to achieve an 
understanding of how human enzymes systems increase or decrease the toxicity of environmental 
toxins. Researchers examine how environmental elements affect the liver, lungs, intestines, nervous 
and immune systems, etc. They also investigate possible carcinogenic, genetic and reproduaive 
effects. Examples of funded projects include a prospective study of diet and cancer in women, a 



Office of Management and Budget, CaUlog of Federal Domestic Assisunce 296 (1989). Sec also 42 U.S.C. §241; National Cancer 
Instiiuu, NCI Fact Book 26 (1989). 

^'CFDA at 296. 

telephone interview with Dr. David Longfellow, Branch Chief, Chemical and Physical Carcinogenesis Branch, Division of Cancer 
Eliology, National Cancer Instituu. NIH. September 28, 1990, Friday, 10:(X) a.m. and Oct. 3. 1990. 1200 p.m. (301) 496-5471 . 

^CFDA at 296. The SBIR program is a federal program that cuu across all of the scientific granting agencies. It was esublished in 
1982 by the Small Business Innovation Ltevelopment Act of 1982. Pub. L. 97-219; Pub. L. 99-158. 42 U.S.C. §285. SBK Phase I 
granu (of approxinuiely 6-month8 duration) are to establish the technical merit and feasibility of a proposed research effort that may lead 
to a commercial product or process. Phase D grants are for the continuation of the research initiated in Phase I and which are likely to 
result in commercial products or processes. Only Phase I awardees are eligible to apply for Phase 11 support. 

^Division of Extramural Research and Training, National Institute of Environmental Health Sciences, Grant Support for Investigator 
Initiated Research Projecu (undated manuscript). 

telephone Interview with Dr. Thor Fjellsledt, Deputy Director, Division of Extramural Research and Training, NIEHS, NIH. 
Thurs. Sept. 27. 1990. (919) 541-7723. 

^*Divi8ion of Extramural Research and Training, National Institute of Environmenul Health Sciences. Grant Support for Investigator 
Initiated Research Projecu (undated manuscript). 

telephone Interview with Dr. Thor Fjellstedt, supra note 25. 
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study of the epidemiology of melanoma, a study of cellular and biochemical effects of tumor 
promoters and experimentaJ tobacco carcinogenesis.^ 

The Program on Biological Response to Environmental Health Hazards has a budget of 
approximately $30,000,000 with which it supports 150-200 research grants. ^^ Grants range from 
$43,000 to $1,544,000, with a median of $175,800. Like NCI, NIEHS awards grants to universities, 
colleges, hospitals, public agencies, nonprofit research institutions and for-profit organizations. 
NIEHS also has a SBIR program similar to that of NCI. Examples of funded grants include 
molecular mechanisms of organophosphate immunotoxicity, the molecular basis for initiation of DNA 
repair, and a probable mechanism for the carcinogenicity of 2-nitropropane. 

3. The General Clinical Research Centers Program 

The Program for General Clinical Research Centers (GCRC) in NIH's National Center for 
Research Resources provides major funding to construct and maintain research centers within 
hospitals at academic institutions to host investigators funded by other NIH components, other federal 
and state agencies and the private sector.*' The purpose of these grants is to create and sustain 
specialized institutional resources in which clinical investigators can observe and study human 
physiology and disease. Centers operate as "discrete, multi-departmental, multi-categorical research 
inpatient and outpatient units." They are intended to be "institutional resources where scientists from 
many departments correlate their laboratory studies with carefully controlled clinical investigations 
and analyses."^' When the number of researchers at an institution receiving outside funding (often 
from NIH) reaches a "critical mass," high level administrators can prepare a proposal for GCRC 
funding. In the words of one investigator, the centers are like a "holding company" for a group of 
principal investigators at an institution. A GCRC award lends prestige to an institution and can help 
it in recruiting new researchers. Grants can be used to pay the cost of "renovation of facilities, 
equipment, hospitalization, core laboratories, salaries of center directors, biostatisticians, nurses and 
technical staff, and other operational costs."'* At present there are seventy-four such centers 
throughout the United States. 

The Program awards approximately $120 million in grants to the 74 grantee institutions and 
several other entities. The grants range from $300,000 to $4 million annually, and average around 
$1.6 million. A small SBIR program awards one or two grants per year. Phase I SBIR grants run 
about $50,000, and Phase II grants can be as large as $3 million. General Clinical Research Centers 
typically support multiple projects from many disciplines, including internal medicine, pediatrics, 
surgery and obstetrics and gynecology. Projects address all aspects of disease mechanisms and may 
range from studies of nutrition and growth and development to genetics and immunology. One 
typical SBIR project supported the development of computer software for handling clinical research 
data.'' The vast majority of the centers are refunded each year, but one or two centers typically drop 
out or are not refunded. Although one might predict that the competition would be quite stiff for the 
two or three available grants, in fact the agency receives only 1-3 applications for new centers per 
year.'* 



™CFDA«t296. 

^CFDAat 186. 

'*>See 42 U.S.C. §287 el «eq. 

"Although gram awards may be made to for-proru institutions, the granu are not intended lo support the routine development of 
producu for sale. For example, it is not proper for a for-profit drug company to use a GCRC funded Center for safety and efficacy testing 
of drugs as part of the application process for drug approval at the Food and Drug Administration without paying the center for the 
resources uacd in the project. On the other hand, if a scientist associated with a Center desires to test a new idea that he has developed 
about the mechanism of the drug, use of the Center's resources would be appropriate. 

'2CFDAat282. 

"CFDAat283. 

'^Telephone interview with. Dr. Bernard Talbot, Program for General Clinical Research Centers, National Center for Research 
Resources, NIH, September 26, 1990. So few institutions apply for funding each year because of the agency's tradition of continuing 
funding for existing grantees and because of the expense of preparing an extensive application for a large center. 
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4. The Biomedical Research Technology Program 

The Biomedical Research Technology Program is another program in the National Center for 
Research Resources of NIH. The program awards grants to assist academic, nonprofit and for-profit 
institutions in developing and sustaining the kind of sophisticated technological capabilities that are 
vital to modem biomedical research. These grants provide biomedical investigators with exceedingly 
expensive and sophisticated tools for research, education and patient care, such as facilities for 
computing and data processing, high-voltage electron microscopy, mass spectroscopy, nuclear 
magnetic resonance spectroscopy and other specialized instrumentation. The program's focus on 
technology, rather than pure research, makes this Program somewhat unique in NIH. Special 
emphasis is given to large scale multidisciplinary projects of regional and national scope. The grants, 
which may be awarded to institutions of higher education, hospitals and other institutions and other 
organizations with programs of biomedical research and specialized research services, provide fiinds 
for primary equipment, professional staff and related operating expenses. Smaller awards are 
available to provide support for pilot projects to demonstrate feasibility of innovative ideas in the 
fields of high technology and engineering." A small number of SBIR grants are also available. 

The Program's annual budget of about $40 million is used to fund 55 biomedical research 
centers, 10-20 resource-related research grants, 10-20 other small grants, and 10-20 SBIR awards.'* 
Grants range from $16,897 - $2,192,567, and average $573,587.3' Average Phase I SBIR awards are 
for approximately $50,000, and Phase II awards may range up to $500,000. ^^ The program is very 
competitive. Less than 40 percent of large grant applications are funded, and only about 25 percent 
of small grant applications receive awards." Examples of funded projects include biomedical 
computer centers, biological structure research, biomedical engineering and biomedical and 
biophysical analyses. Examples of funded SBIR projects include research aimed at producing 
interactive instructional videodiscs and developing fiber optic sensors. ■" 

B. Criteria for Awarding Grants 

NIH can award individual research grants to any nonprofit or for-profit organization, company 
or institution engaged in biomedical research. The usual beneficiaries are universities and nonprofit 
research institutions. Applicants for GCRC grants must be medical schools, research hospitals or 
similar institutions. SBIR grants can only be awarded to domestic small businesses.*' The broad 
criteria for awarding grants are the same for all of the Institutes: The major elements in evaluating 
proposals include assessments of: (1) the scientific merit and general significance of the proposed 
study and its objectives; (2) the technical adequacy of the experimental design and approach; (3) the 
competency of the proposed investigator or group to successfully pursue a project; (4) the adequacy 
of the available and proposed facilities and resources; (5) the necessity of the budget and components 
requested in relation to the proposed project; and (6) the relevance and importance to announced 
program objectives.*^ Although the criteria for awarding General Clinical Research Center grants 



"CFDA«t29I. 

^FDA at 292; CFDA update at E-55. 

'^A $650,000 per year limit is now in eflect except where exceptional justifications can be made. CFDR at 292. 

^CFDA at 292. 

"CFDA at 292. 

*CFDA at 292. 

*'SmaIl businesses are entities that are independently owned and operated for profit, are not dominant in the field in which research is 
proposed, and have no more than 500 employees. The snull business entity must constitute the primary employment (more than one-half 
time) of the principal investigator at the time of award and during the conduct of the proposed project. According to one NCI grant 
administrator, there are two categories of typical applicants: 1) individuals formerly associated with large institutions who are very familiar 
with the program (these applicants are usually successful); and 2) others with some training, usually in software development who keep 
applying and finally, by trial and error, get funded. 

*^FDA at 283. The primary criteria for evaluating Phase I SBIR grant applications include: (I) the technical merit of the proposed 
research; (2) the soundness of the proposed design and methods; (3) the qualifications of the proposed principal investigator, supporting 
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vary somewhat because of the focus on funding centers for multiple research efforts,*^ the evaluators 
tend to apply the above criteria to the research that is to be carried out at the centers in deciding 
whether to award a grant to the center itself. 

C. Procedures for Awarding Grants 

NIH must by law "require appropriate technical and scientific peer review of . . , applications 
made for grants and cooperative agreements ... for biomedical and behavioral research,"** and peer 
review plays an exceedingly important role in determining which projects are funded. Applications 
for NIH support go through a "dual review" process. A grant application is assigned according to 
subject matter by NIH referral officers to one of 90 standing peer review groups, usually referred to 
as "study sections," that are located within the Division of Research Grants. ** 

The Division of Research Grants, which houses the study committees, is not within any of the 
individual institutes, and its staff does not report to any of the Programs that are responsible for 
funding and managing the grants. Each study section has a "Scientific Review Administrator" 
(formerly called "Executive Secretary") from the staff of the Division of Research Grants to provide 
clerical and ministerial support. Approximately 2700 external reviewers serve four-year terms on 
NIH study seaions.** They are paid $150 per day for their efforts plus travel expenses and per diem. 
Potential peer reviewers are identified through several sources, including the committee's Scientific 
Review Administrator's knowledge of the field and the scientists who work in it, the Scientific 
Review Administrator's contacts at scientific meetings, NIH staff recommendations, existing research 
grant applications and awards, research publications, and recommendations of existing panel 
members.'*'' No study section may have more than one member from the same institution. Although 
most study section members are appointed for fixed four-year terms, members can be added on an ad 
hoc basis when additional expertise is needed to evaluate particular proposals. In addition, when no 
existing study section appears to have the requisite expertise to evaluate a proposal or group of 
proposals, an ad hoc study section can be appointed for the limited purpose of reviewing a small 
number of applications.* Ad hoc committees are also used to evaluate applications from other study 
seaion members. Most SBIR applications are reviewed by ad hoc committees, and the Programs 
make a special effort to locate scientists in industry for membership on these committees.** 

Each study section typically holds three annual meetings'" at which the reviewers collectively 
apply the Program's previously articulated selection criteria to the grant applications assigned to the 
section. When no member of the panel has sufficient expertise to evaluate all aspects of a proposal, 
one or more external written reviews may be requested to aid the panel.*' The application and any 
outside reviews are assigned to two or more panel members (called the primary and secondary 

■ufT and consulunU; (4) the potential of the proposed research for technological innovation and commercial application; (S) the 
appropnaleness of the budget requested; and (6) the adequacy and suilabiliiy of the facilities and research environment CFDA at 186 

^^Institutions must denxtnstrau potential research productivity, quality of the proposed physical facility, and soundness of the 
administrative plan for the cenur. CFDA at 282. 

**Health Research Extension Act of 1985 f492(a)(l){A). 42 U.S.C. §289a. 

^United Sutes General Accounting Office, University Funding: Information on the Role of Peer Review at NSF and NIH 14 (March 
1987), at 15 [hereinafter cited as 1987 GAG Report). The National Cancer Institute and the National Institute of Neurological and 
Communicative Disorders and Stroke in 1985 initiated grant programs designed to uke special account of the applicants' track records in 
making longer-term awards. These special programs have relied on nuil reviews similar to those typically used in NSF. See Culliton, 
NIH Proposes Extending Life of Granu. 226 Science 1400 (1984). 

**! 987 GAG Report, aupra note 45. at 15. 

*^ld. at App. IV. 

*Some programs make more extensive use of ad hoc study sections than others. For example, most applications in the Biomedical 
Technology Program are reviewed by ad hoc committees, although there are a few permanent study sections. 

'^ this regard, NIH differs from NSF, which declines to place scientists from industry on SBIR review committees because of the 
potential for conflict of interest. 

^tudy section meetings arc usually held at hotels near the NIH campus or at the locations of important scientific meetings. 

*'NIH Manual 4510, "Referral and Initial Review of NIH Grant and Cooperative Agreement Applications 9 (July 1. 1982). 
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reviewers) who prepare detailed written reviews prior to the panel meeting at which the application is 
discussed. All panel members are expected to read all of the applications." Study sections in 
programs, like the General Clinical Research Center and Biomedical Research Technology programs, 
that fund large capital projects, conduct site visits to the institutions that house the projects at least 
one time per grant cycle. There are usually about 15 visitors, nearly all of which are from academia. 
Applicants in these programs are given the opportunity to specify particular scientists that they prefer 
not be on the panel conducting the site visit." 

The study section's evaluation of each application is typically broken down into two separate 
steps. After hearing from the primary and secondary reviewers and discussing the proposal, the 
group first decides whether the application is deemed "worthy" of funding based on the NIH selection 
criteria. Approximately 90 percent of the applications pass this preliminary test.** If an application 
is deemed worthy, the study section next assigns it a priority rating from which the NIH staff 
computes a priority score. A "summary statement," which is prepared by the Scientific Review 
Administrator from the two pre-meeting written reviews and his or her notes of the meeting, relates 
the application's score and articulates the group's reasons for assigning it that score." The 
application files are then sent to the Program Directors for the relevant Programs in the Institutes. 
Only the summary statement accompanies the application to the next stage of the review process- 
review by the Advisory Council for that Institute.^ 

Each Institute has an Advisory Council composed of scientists and lay persons appointed by the 
Secretary of Health and Human Services or, in some cases, by the President. Membership on some 
Advisory Councils is regarded as a political plum; for example, the lay members of the National 
Cancer Advisory Board (the Advisory Council for NCI) are usually prominent supporters of the party 
in power that have an active interest in cancer.'^ In addition to providing broad advice on how the 
Institute should allocate funds among programs, the Advisory Councils review the summary 
statements, and recommendations of the study sections for scientific merit and for consistency with 
nontechnical programmatic goals.** 

Although it would probably not be accurate to describe Advisory Council review as perfunctory, 
the Councils are highly deferential to the study section recommendations.*' Only about ten percent of 
the applications are singled out for review by the Advisory Councils, and a much smaller proportion 
result in any special action by the Councils." The Councils almost never involve themselves in 
minor adjustments to the budget, and they tend to avoid detailed involvement in the nitty gritty of 
detailed scientific evaluation. Their purpose is to evaluate the overall output of the study section 
meetings, not to second guess the study sections on individual applications. If a Council is 
dissatisfied with a study section's output, it can remand one or more matters to the study section for 



*2ld., at 9. 

*^elephone Interview with Dr. Bernard Talbot, supra note 34. 

**Chubin & Hackett, supra note 3. 

**The summary statement, 'while not necessarily exhaustive, should be complete and represent a group evaluation of each 
application.* It must contain "objective factual information, carefully documented and well justified* It must also include "pertinent 
material from the meeting discussions.* NIH Manual 4510, supra note 51, at 13. 

**See NIH Manual 4510, supra note 51, at 9; Chubin & Hackett, supra note 3, at 20-21; 1987 GAG Report, supra note 45, at 1 1-13. 

*^Past members of the National Cancer Advisory Board include Ann Landers, Richard Block of H & R Block, and former United 
States Senator Tim Lee Carter. 

*^rhe Secretary chooses Advisory Council members from among nominees from NIH, members of Congress, special interest groups, 
other entities within HHS and the general public. 1987 GAG Report, supra note 45, ai 16. For example, the Advisory Council for the 
General Clinical Research Center Program has 18 members. Six are public members who are individuals with other expertise in law, 
economics, math, and public policy. Twelve are scientists selected on basis of scientific expertise. All are appointed by HHS 
headquarters. 

^'Chubin and Hackett conclude that: The NIH awards process is hardly one of 'dual review* for the second stage rarely considers the 
merits of individual proposal (or, if they do so their consideration nukes essentially no difference in the final rankings).' Chubin & 
Hackett, supra note 3 . 

"1987 GAG Report, supra note 45, at 13; Chubin & Hackett, supra note 3, at 22. 
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reconsideration. This limited Advisory Council involvement is, in any event, an inevitable 
consequence of the program's workload. For example, in the National Cancer Institute, the agency 
staff mail about 1400-1500 summary statements out to Advisory Committee members 2-3 weeks 
before the triermial meetings. Even the most conscientious Council member cannot realistically 
examine all of the evaluations in detail prior to the meetings. 

From the Advisory Councils' comments, the applications, and accompanying summary 
statements, the Institute's Program Director ranks the "fundable" applications. Because scientific 
merit is not the only criterion, the staff is permitted to depart from the ranking suggested by the study 
sections and approved by the Advisory Councils. Yet this happens onJy extremely rarely O^ss than 
three percent of the time),«' because it requires the Program Director to draft a special petition and 
pursue the matter through the Institute's hierarchy. Some, but not all. Program Directors typically 
set aside some funds to allow the directors the discretion to fund the projects that fall very close to 
the cut-offline, a process that is referred to internally as "ftinding by exception. "^^ 

NIH has always interpreted the Privacy Act to require that its peer review files for individual 
grant applications be made available for examination and correction by the applicants but otherwise 
shielded from public disclosure. Portions of panel and Advisory Committee meetings that discuss the 
merits of individual proposals are also closed to the public.® Panelists are instructed that all 
materials related to the review of grant applications are strictly confidential, and no written materials 
may be removed from the conference room. Reviewers may not share with applicants or their 
institutions the content of the panel deliberations.** No particular sanctions are specified, however, 
for breaches of confidentiality. In practice, the only sanction is the stigma that accompanies removal 
from the panel. 

NIH makes grant applications for funded projects and a general description of the awards 
available to the public after each funding round has been completed. Interim and final progress 
reports and the results of any audits or reviews of the grant are also routinely made public. Pending 
or disapproved applications for new grants, memoranda and transcripts from panel meetings, and 
other written communications from reviewers are not automatically made public, but information 
contained therein may be released upon a request pursuant to the Freedom of Information Act 
(FOIA)." Although the D.C. Circuit Court of Appeals held in Washington Research Project, Inc. v. 
Dept. of Health, Education and Welfare" that the contents of all grant applications were generally 
releasable under FOIA at the end of the funding cycle, NIH has recently taken a very broad view of 
the applicability of exemption 4 of that statute to information contained in grant files. Exemption 4 
allows agencies to withhold information that would disclose "trade secrets and commercial or 
financial information obtained from a person and privileged or confidential."*^ NIH believes that, 
given the recent trend toward commercialization of government-sponsored research, a good case can 
usually be made for withholding much of the contents of a grant application under this exemption. 
The validity of this argument has yet to be tested in the courts.** NIH further takes the position that 
information contained in summary sheets (including priority scores), transcripts and summaries of 



*' 1987 GAG Report, supra iKHc 45, at 13. 

**relephonc inlerview wiih Dr. Iris Obrams, Branch Chief for Extramural Programs, Epidemiology and Biosutisiics Division, NIH. 
Friday. November 2, 1990. 

^'NIH Manual 4513 (1982), at 2. 

**NIH Manual 4510, supra noU 51. at 7. 

*^lJnited States Public Health Service, PHS Granu Policy Statement 15 (1987). 

**504 F.2d 238 (D.C. Cir. 1974), cert, denied. 421 U.S. 963 (1975). 

•"5 U.S.C. §552(b)(4). 

*^The argument has a surface implausibility. If the applicants are willing to have the information in grant applications viewed by peer 
reviewera, some of whom nuy be direct competitors, it is hard to sec how it could seriously be nuinlained thai the information is 'trade 
secret." See generally McGarity and Shapiro, The Trade Secret Sutus of Health and Safety Testing Information: Reforming Agency 
Disclosure Policies, 93 Harv. L. Rev. 837 (1980). 
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peer review meetings and outside reviews that constitutes opinion, rather than purely factual 
information, comes within exemption 5 of FOIA, which applies to internal agency communications. ** 

The printed forms for initial applications inform all potential applicants of their Privacy Act 
rights. Pursuant to that statute, NIH routinely provides all applicants with staff-prepared summaries 
of both written reviews and panel minutes (with priority scores attached) after the completion of the 
panel review process and prior to submittal of the application to the relevant Advisory Council.'" 
NIH regards this as an important source of feedback for future applications and resubmittals of 
declined applications." If an applicant requests copies of the actual reviews (called pink sheets 
because of their historical color) and minutes, they are sent with certain information (e.g., the 
identities of the reviewers and panelists) redacted. "^ Transcripts of panel meetings and the actual 
minutes of the meetings are not made available. For the most part, applicants seem willing to trust 
the staff summaries and do not request original documents.^ The applicants have an opportunity to 
rebut the conclusions reached by the study sections in submissions to the Advisory Councils, but 
these result in changes only rarely and only in cases of factual errors critical to the decision.'" 
Applicants are informed of the identities of peer review group members on their pink sheets, but the 
identities of the panelists who wrote the primary and secondary written reports to the study section 
and the identities of the authors are not made available to the Program Directors or the applicants, 
and their written and oral comments are not otherwise revealed to applicants or the public." 

Each potential reviewer must fill out a financial disclosure statement that reveals the extent of 
that person's financial interests in various institutions and companies. NIH conflict of interest 
regulations and guidelines prevent a panel on which a reviewer sits from considering the reviewer's 
application or an application from the reviewer's "spouse, parent, child, partner, or close 
professional associate." In addition, a panelist must leave the meeting when the panel is considering 
an application from his or her own organization or an organization with which he or she is 
negotiating for future employment."^ The guidelines urge all reviewers to avoid the "appearance" of 
a conflict of interest by refraining from participating in the evaluation of applications involving a 
recent student, a recent teacher or a close personal friend. Finally, reviewers "should not participate 
in the review of an application from a scientist with whom the reviewer has had long-standing 
differences which could reasonably be viewed as affecting objectivity."^ The Director of NIH may 
waive these guidelines and prohibitions "if he or she determines that there is no other practical means 
for securing appropriate expert advice on a particular grant application . . . ."™ Except for a terse, 
but foreboding reference to "relevant provisions of Title 18 of the United States Code, relating to 



5 U.S.C. §552(b)(S) (exemption for 'inter-agency or intra-agency memorandums or letters which would not be available by law to a 
party other than an agency in litigation with the agency'). 

^M Manual 4512 (1982) as superseded by NIH Instruction and Information Memorandum CERT 83-4, December 21, 1984. See 
also Memorandum to Distributees from Associate Director for Extramural Research and Training, NIH, dated August 24, 1978 re: Routine 
Release of Summary Statements following National Advisory Councils and Boards; When the Advisory Council ukes a position difTereni 
from that recommended by the review panel, the package to the applicant must include a letter indicating the Council's decision and a 
supporting rationale. Id. 

^'NIH Manual 4512 (1982), ai 4. 

^United Sutes General Accounting Office, Peer Review: Compliance With the Privacy Act and Federal Advisory Commituc Act 7 
(1991) [hereinafter cited as 1991 GAG Report]. 

'^Id., at 7. NIH receives fewer than 10 Privacy Act requests per year. Id. 

"For example, reviewers of a recent application were favorably impressed by the project, bui were concerned that the principal 
investigator had conducted research for four years without a single publication resulting. After the committee recommended that the grant 
not be funded, the applicant wrote a rebuttal letter explaining that he had had been in medical school during the relevant time period had 
had actually conducted research only in the summers. The application was later funded 

■'*NIH Manual 4514 (1982). at 6. 

^^42 C.F.R. §52h.5(b). Sec also NIH Manual 1805 6-7 (1982). 

""NIH Manual 4510, supra note 51. at 7; NIH Manual 1805 at 6-7. 

^^42 C.F.R. §52h.5(c) (1990). 
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criminal activity," the regulations and guidelines do not provide any sanctions for breach of their 
provisions.*^ 

Applicants have a limited right to request reconsideration of panel determinations prior to 
consideration by the Advisory Councils." Staff-prepared review summaries (the pink sheets), with 
priority scores attached, are sent to all applicants after the panel meetings. Pursuant to the Privacy 
Act, applicants may demand that the agency correct any information that is not accurate, relevant, 
complete or timely prior to submitting the applications to the Advisory Council. The rejected 
applicant first consults with the Program Director in charge of the application who refers any written 
rebuttals to the primary and secondary reviewers to correct any factual errors in their evaluations of 
the proposal. Obvious factual errors can be corrected before the group of applications goes to the 
Advisory Council. Rejected applicants may also argue that the study section did not contain any 
scientist with sufficient expertise in the relevant scientific field. NIH takes the position, however, 
that "(mjatters of expert opinion are not subject to amendment" in light of Privacy Act responses. 

When the Program Director is persuaded that the study section made a mistake on a matter of 
opinion, such as the overall importance of the project, the staffer will urge the applicant to resubmit 
the application to a different study section or request a referral officer in Division of Research Grants 
to appoint an ad hoc study section, rather than continue to resubmit to the same committee. If the 
staffer is not convinced that a mistake was made but thinks that the project deserves funding, he or 
she will usually advise the rejected applicant on ways that the application can be improved for 
submission in a future funding cycle. For example, the staffer might suggest a collaborator whose 
inclusion in the project might enhance its chances for funding in the future. 

The staff makes the Advisory Councils aware of any rebuttals that applicants file. Rebuttal 
letters can have an impact on the Council's rankings in cases of miscalculations or misinterpretations 
of a researcher's qualifications, but the Councils do not normally pay attention to differences of 
scientific opinion. Unless requested in advance, the Advisory Councils only have the summary 
statements before them, and they are disinclined to second guess the experts who have spent more 
time evaluating the merits of the proposals. Most of the scientist members on the Advisory Councils 
were previously members of study sections, and they know that too many reversals could have an 
impaa on the willingness of study section members to serve in the ftiture. 

After the relevant Advisory Council has completed its review of the proposals, mid-level NIH 
staffers (usually Branch Chiefs) ascertain from the agency's budget office the rough percentage of 
proposals that may be funded and meet together to draft final recommendations to the Institute's 
Director. At these meetings, the Branch Chiefs can compete with one another for additional funding 
"by exception" from the previously mentioned pool of ftinds set aside at the outset. The Branch 
Chiefs can change the ranking of proposals at the margins. The goal of these meetings is to look at 
the broad picture and present the Director with a recommendation that is defensible to the outside 
world, which includes rejected applicants and sometimes interested congresspersons. The final 
decisions are usually made by executive committees composed of the Institute's Director and the 
directors of the various divisions within the Institute. 

NIH has promulgated procedures for challenging initial review recommendations. The agency 
recognizes that applicants may appeal decisions not to fund for several reasons, including "perceived 
insufficient expertise on the initial review group ... or conflict of interest on the part of one or more 
of its members; apparent factual or scientific errors, oversights, or bias associated with the review of 
an application at the initial or advisory council review; and possibly inappropriate handling of the 



^42 C.F.R. §52h.5(a) (1991). 

^Applicants must follow formal rebuttal procedures. They are not allowed to communicate with Council members concerning funding 
matters. Council members are provided a standard form to nil out if applicants ever attempi to communicate wiih them about funding 
matters. This sort of conuct almost never happens in NIH. 
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review or of the application."*' The appeals process, however, "is not intended to resolve purely 
scientific disputes between peer reviewers and the investigator; to provide a mechanism for allowing 
investigators to submit information that should have been presented in the original proposal; or to 
provide a forum for disputing priority score determinations in the absence of specific and substantive 
evidence pointing to a flawed review, "^^ In practice, appeals are usually only taken in cases of 
terminations of ongoing grants. Appeals must be taken to the Review Officer in the Office of the 
Director, but initial review panels and Advisory Councils will usually be asked to participate in the 
resolution of the appeal. Actions that may be taken in the case of valid appeals include "rereview by 
the same or another initial review group; special consideration by the advisory council; or 
administrative action authorized by the Institute Director or staff."" There is no procedure for 
appealing final funding decisions. 

D. The Process in Operation 

The peer review process at NIH is often held out as a model of how peer review ought to be 
carried out in awarding discretionary grants. Yet although the NIH peer review process is generally 
highly successful, problems occasionally arise. The following discussion addresses some of the 
issues that NIH staffers, applicants, reviewers and outside commentators have identified. While the 
following discussion necessarily tends to focuses on criticisms of the programs, support for the NIH 
peer review system as a whole is both broad and deep among those who deal with it on a regular 
basis. 

1. Feedback and Rebuttal 

Unlike some research granting institutions, NIH provides applicants with a statement of reasons 
in the form of a pink copy of the evaluation of the peer review committee. This can be especially 
useful to researchers who are not funded, because researchers can use the pink sheets to identify 
improvements that can be made in future applications. Applicants cannot, however, gain access to 
the actual minutes or transcripts of the panel meetings. Indeed, the pink sheets have been sanitized to 
a limited extent by the Program Directors who typically return to the Scientific Review 
Administrators any summary statements containing derogatory statements and obvious factual errors. 
Even after sanitization, the pink sheets can sometimes reveal indications of bias among one or more 
members of the panel. 

Rejected applicants can take the pink sheets to the Program Directors and explore the possibility 
of preparing a grant application for a different program where it will have a higher probability of 
success. If the Program Director is persuaded, he or she will "lobby" the staff of the Division of 
Research Grants to assign the next proposal to a different study section. Most of the researchers 
interviewed for this report were of the opinion that the agency staff was very helpful in explaining 
why applications were rejected and in making suggestions for improvement or for sending proposals 
to different study sections.*^ On the other hand, there is a sense among some applicants that the 
explanations that the peer reviewers provide are post hoc rationalizations. Sometimes rejected 
applicants perceive that the real reasons for poor reviews are not provided on the pink sheets and may 



NIH Instmclion and Informaiion Memorandum CERT 85-2, dated March 19, 1985; see also NIH Guide for Grants and Contracu, 
Vol. 14, No. 4, March 29, 1985. 

NIH Instruction and Information Memorandum CERT 85-2, supra note 81 ; see also NIH Guide for Grams and Contracts, supra note 
81. 

'^NIH Instruction and Informaiion Memorandum, supra note 81. 

^^One researcher related that after his first application was not funded, he called the scientific review administrator for the commitue 
that reviewed his application, and they went over the pink sheet on the telephone. The scientific review administrator also read the minutes 
of the study section meeting and told the researeher that the panelists were interested in funding the proposal if some improvements were 
made. He resubmitted the proposal in the next funding cycle, and it was funded. As a result of this experience the researcher learned to 
ask NIH staffers: 'Was the proposal appropriate? Did it go to the right committee? Were they interested?" Telephone interview with Dr. 
Raymond Novak Director of Institute of Chemistry & Toxicology and Professor of Pharmacology, School of Medicine, Wayne Slate 
Univereity, March 3, 1991 - 4:35 p.m., 313-577-0100. 
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in fact be relatively trivial. One applicant interviewed for this report was persuaded that the 
comments on his pink sheet were designed to steer his research in more conventional directions that 
the committee believed were more consistent with their own research perspectives. 

Ironically, because it is willing to share summary statements with the applicants, NIH invites 
challenges based upon factual inaccuracies that inevitably creep into the written reviews. An 
applicant justifiably feels unfairly treated when the Institute's explanation contains factual errors, 
even if the project probably would not have been funded in the absence of the errors. NIH provides 
applicants an opportunity to cure any errors on the pink sheets prior to submission to the Advisory 
Councils, but given the limited substantive role that the Advisory Councils play, it is highly unlikely 
that rebuttals will affect the rankings at that level. One researcher interviewed for this project 
suggested that applicants be given an opportunity to correct obvious errors on the pink sheets before 
the peer review seaions make their final recommendations. This would not necessarily require an 
additional meeting of the study section. Like motions for rehearing in court, the rebuttals could be 
circulated to members of the study sections by mail. If more than a predetermined number of the 
study section members voted to rerank the applications in light of the rebuttal, this could be 
accomplished via a conference telephone call. 

NIH staffers argue that much the same result is accomplished through the informal process that 
has evolved whereby a disappointed applicant may submit a written rebuttal to the Program Director 
for presentation to the Advisory Council. Even though the Advisory Council will not likely change 
the priority scores, it can note its opinion that a mistake was made, and the Program Director can cite 
that fact at the Branch Chief level in arguing that the proposal should be funded by exception. NIH 
staffers further observed that even if rejected applicants are only given a relatively brief period in 
which to submit rebuttals, the process could easily become bogged down if a significant percentage of 
the more than 30,000 annual applicants exercised a rebuttal right. 

2. Favoritism 

The NIH staffers interviewed for this report were generally confident that scientists could put 
aside personal friendships and vote on the m'^rits of the applications before them. Nevertheless, NIH 
attempts to avoid even the appearance of personal favoritism by excluding from study sections 
applicants, family of applicants, and their co-workers both at their institutions and at other 
institutions. In addition, although NIH's conflict of interest regulations allow a person to sit on a 
panel that considers applications from the panelist's institution, the panelist must leave the meeting 
when the panel is considering an application from his or her own organization or an organization 
with which he or she is negotiating for future employment. Unlike a similar provision in the 
National Endowment for the Arts' regulations," this aspect of the NIH conflict of interest regulations 
has been relatively uncontroversial. 

3. Old Boy Network 

Like virtually all peer review-based granting organizations, rejected applicants frequently 
characterize the NIH process as an "old boy" network in which those who are already in the process 
dominate the peer review panels and parcel out the awards among themselves. For example, when 
one frequently funded researcher decided to shift his research interests to a slightly different area, he 
was told by friends high up in NIH that the field in which he proposed to conduct research was a 
"club" and he was not a member of the club. They subtly suggested to him that he would enhance 
his chances of receiving a grant in the new field if he hired a member of the club as a consultant on 
his project."* Another suggested technique is to offer to make a member of the committee (preferably 



"See Pan 5, infra. 

'*One NIH staffer suggested that recommendations that esublished applicanu find collaborators when they desire to enter new fields is 
less a matter of 'old boyism* than a fear that the proposal will otherwise lack sufficient expertise. 
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the chairman) a co-author of a paper as an introduction into the old boy network.*^ Still another 
technique for new researchers is to circulate drafts of the grant proposal to prominent scientists in the 
field who are likely to be members of the study sections for comments prior to submitting the 
proposal to NIH. The recipients will feel flattered that their advice was requested, and they will 
obtain a familiarity with the subject matter of the proposed research quite apart from their status as 
members of the panel.** 

While recognizing the fact that it is often hard for a young or inexperienced researcher to break 
into the system by securing an initial grant, one NIH staffer attributed this to the novice's general 
lack of familiarity with the procedures and with the evaluative criteria, rather than to any favoritism 
for the members of the "club." The staffer also noted that NIH committees already contain a fairly 
large number of Assistant Professors with little prior experience with the NIH grants process, thus 
reducing the chance that a given conmiittee will be dominated by an "old boy" faction. Finally, NIH 
staffers observe that many established scientists do not get funded and many newcomers do get 
funded. 

4. Halo Effect 

Closely related to the "old boy network" problem is the "halo effect" through which a well- 
established researcher is funded for projects that are not especially well designed or that are not in a 
priority area merely because of the principal investigator's reputation as a pillar of the relevant field. 
NIH staffers generally believe that, like seeding in professional tennis, a scientist should be evaluated 
on the basis of his or her recent performance, not on the basis of past reputation. But they all 
recognize that scientists who are regarded as the leaders in their fields get "special consideration" 
unrelated to the merits of their proposals. One staffer related an experience in which a very 
prominent investigator submitted an application for a continuation of an existing grant that reflected 
major modifications in the identity of the researchers and in the project's scope with little explanation 
for the changes. The Program Director was upset by the investigator's implicit assumption that he 
had an entitlement to funding, and he arranged a meeting with the investigator to "sort of haul him 
over the coals." Before the meeting, the investigator won a Nobel prize "and let me tell you, the 
meeting was immediately canceled." Another staffer noted that it is very difficult to refuse funding 
to a scientist who has won a Nobel prize, even when his applications are "kooky" and everyone 
involved recognizes that he is past his prime. 

Although the halo effect may have distorted the process in the past, it is not clear how large a 
role it plays in the current age of very tight budgets. One interviewee explained that reviewers are 
increasingly unwilling to keep an old timer afloat with funds that could be spent on more relevant or 
productive projects. Another suggested that NIH staffers are disappointed when an established 
scientist submits a poor proposal, and they subtly suggest that the proposal be withdrawn and 
amended. If a peer review panel knows that an application from a very highly regarded researcher is 
below par, the panel can limit the duration of the grant as a "slap on the wrist." 

5. Personal Animus 

When research support monies are in short supply, the animus of a single member of a study 
section can deprive a good project of support. Since Uie scores of the panel members are typically 
averaged together, an especially low score from one panel member can be outcome-determinative. 
One frequently funded researcher interviewed for this project was convinced that one of his projects 
failed to obtain funding because the review panel contained a "wild man" who "ranted and raved" 
about the proposal and gave it a low score. Although panel members are supposed to be anonymous, 
this researcher found out that the panel member had done some parallel work in the past that had 

*^ According to NIH stafTers, this strategy would be of only linuled utility, because co-authors are not eligible to review the application 
of a collaborator for a period of lime after publication of a co-authored paper. 

**NIH staffers also questioned this strategy, because the agency's informal rules of practice require a reviewer to excuse himself if he 
has been asked to review a proposal ouuide of the forma! NIH process. 
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reached results that varied from those of the applicant's recent research. TTie project was later 
recommended for funding by a section with a different membership. 

The fact that rankings are made in meetings in which as many as 25 scientists are present helps 
shield against animus on the part of any single committee member. Each participant knows that the 
other panel members are also experts and will probably detect attempts to grind any axes. One 
frequent panelist noted that, given the experience of the other members of the committee, it is very 
difficult credibly to downgrade a worthy proposal. 

6. Mavericks 

It is difficult in NIH for "mavericks" who advocate unorthodox research proposals to get funded, 
unless they already have a reputation for success. One NIH staffer observed, with an obvious 
reference to the work of Thomas Kuhn,"* that each scientific field has its own "paradigm, world 
view, or set of coordinates" through which researchers see the universe. Several researchers and 
members of the NIH staff observed that projects well within the "mainstream" of previously funded 
research have a much better chance of being funded during any given year than research that rejects 
the conventional wisdom. In fact, in many programs, most funded research consists simply of 
incremental extensions of existing research into slightly different areas. One scientist interviewed for 
this report observed that the existing funding system strongly encourages this sort of cautious 
incremental ism, which he referred to as "me too" science. The result is "a big fraternity of scientists 
all reaching the same conclusions and not testing each others' hypotheses." 

When the scientists from the mainstream control the purse strings, it can be difficult for the 
mavericks to obtain the resources necessary to prove or disprove their out-of-the-ordinary theories. 
One investigator observed that advocates of new or different approaches face a "Catch 22" in the peer 
review process, because the reviewers often criticize proposals on the ground that the kind of 
research outlined in those proposals lacks a sufficient predicate in previously published research. 
Another researcher complained that although his group had published many papers in peer reviewed 
journals, it had difficulty getting NIH grants "because the panels are representative of the opposite 
school of [thought], and it's very threatening for a scientist to see a new idea. It's automatically shot 
down." Several scientists and agency staffers noted that a decision to fund a maverick proposal is to 
some extent an admission that the dominant paradigm may be wrong. If the maverick is right, then 
the case for continued funding of existing grants is weakened." 

One investigator interviewed for this Report was associated with a prominent cancer research 
institute, but held to an unorthodox theory for the mechanism of chemical carcinogenesis. His 
applications to NCI were reviewed by panels composed of adherents to the traditional theories, and 
they were predictably rejected. Rather than opt out of the federal funding system as many mavericks 
do, this researcher complained to the Director of NCI and the Director of the Research Grants 
Division that his projects were being reviewed by biased panelists, and he demanded that his projects 
be reviewed by panels in which adherents to the traditional theories composed less than one-half of 
the panel. After the Research Grants Director acquiesced in this request and appointed a special 
study section, the researcher's applications started to get funded. Another researcher reported that 
after his applications were rejected several times, he strongly suggested the names of people who 
would, in his opinion, objectively evaluate his proposal. Since one of these persons was already on 
an existing study section, the NIH staff allowed the proposal to be resubmitted to that section, and it 
received better scores. 



*^Kuhn, supra note 13. Kuhn observed that advocates of paradigm shifts in a Held are not well received by their peers who adhere to 
the old paradigm. 

^Although a scientist may not sit on a panel that reviews his own gram application, the scientists who agree to sit on peer review 
panels do so partly on the understanding thai their gram applications will have a good chance of being funded by the panel that reviews 
them. The members of the panel are always made aware of the fad that an applicant sits on one of the NIH panels when his application 
comes up for consideration. 
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A maverick's chances of getting funded are substantially enhanced if an NIH staffer can be 
persuaded to take up his or her cause before the Advisory Council. Although the resistance to the 
maverick proposal is still high, the staffer can argue persuasively that Einstein would probably not 
have been funded by a panel of his peers in the early twentieth century. 

7. Tunnel Vision 

Even when the applicant is not clearly a maverick challenging the dominant paradigm, study 
sections can become afflicted with "tunnel vision." One NCI staffer offered as an example a proposal 
for a novel technique for detecting the presence of certain toxic chemicals in the environment cheaply 
and accurately. The study section that reviewed the proposal did not believe that the project could 
advance scientific knowledge and gave the study low priority. The members of the study section 
failed to see the potentially large impact on human health that a cheap and effective exposure- 
avoidance device could have, despite the fact that it would not provide additional information about 
the chemical itself." A researcher interviewed for this Report related that his project to determine the 
effects that various components in diesel fuels have on the general population was not funded by NIH 
because although the reviewers were interested in the individual components of the fuel, they were 
not concerned with human population applications. The researcher complained: "They are never 
interested in projects of this sort that are highly applied in nature, nor are they interested in 
performing risk assessments."'^ 

8. Applicant Anonymity 

NIH makes no attempt to shield the identities of applicants from reviewers, and the NIH staffers 
interviewed for this report were generally opposed to the idea, arguing that the applicant's experience 
or "track record" was an important aspect of the evaluation that could not be evaluated anonymously. 
In NIH programs that manage a small number of very large grants, applicant anonymity is 
impractical and it is, of course, impossible to do an anonymous site visit. Even for some of the large 
programs in the National Cancer Institute, agency staffers believe that anonymity would be very 
difficult to maintain. Reviewers who are generally familiar with the literature should be able to guess 
the identity of the applicant from the topic of the application. 

9. Reviewer Anonymity 

NIH shares a great deal of information and analysis with applicants, including the identities of 
the members of the study sections that review their proposals. Although the identities of the primary 
and secondary reviewers are not revealed, it is often relatively easily deduced by the applicant from 
an examination of the areas of expertise of the panel members. As one Program Director observed, 
"You don't need to be an Einstein to figure out who probably reviewed your proposal." 



"The Btory, however, has a happy ending. The sufTer in charge of the project referred the applicant to the National Institute for 
Occupational Safety and Health, and the study was later funded. 

A similar experience was related by an applicant for NIH funding: 

1 got funding from EPA in FY86 for [a project] to determine which heavy metals found in the environment are toxic to 
the human immune system by screening, i.e., by testing the effects of many meuls whose effects are not known. . . . 
My proposal for this work had been unsuccessful at NIH for three years in a row. This research was 'more applied' than 
ahe kind of projecu that generally receive favorable evaluations at NIH. In addition, many people on NIH Study Sections 
are not very interested in enviroiunental problems. NIH is interested in more basic research to determine the mechanisms 
by which human systems are affected by and respond to external factors. . . . EPA tends to want to know what's 
happening; NIH wants to understand the mechanisms by which it happens. 

American Management Systems, Inc., An Evaluation of EPA's Exploratory Research Grant Program E-5 (1988) (statement of David 
Lawrence, Albany Medical College) [hereinafler cited as 1988 AMS Report]. 

^^1988 AMS Report, supra note 91, at E-6 (statem]ent of Dr. Herbert Rosenkranz, Case Western Reserve University School of 
Medicine). This story also has a happy ending, because the study was ultimately funded by EPA. Because EPA funds a much lower 
percenuge of applications than NIH, however, not all such stones have happy endings. 
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The contents of panel deliberations are supposed to be kept strictly confidential. In practice, 
however, there have been several serious breaches of confidentiality in some of the programs. Many 
NIH staffers and outside researchers interviewed for this project reported instances of breaches of 
confidentiality. A scientist who engages in a relatively narrow field of research can sometimes guess 
the identities his or her primary and secondary reviewers and can press study section members at 
scientific conferences for information about what happened at study section meetings--"Did Dr. X try 
to deep six my grant?" In addition to breeding ugly confrontations in the hallways at scientific 
meetings, NIH officials fear that breaches of confidentiality will inspire an applicant to retaliate 
against (or reward) scientists who provided negative (or positive) reviews of his or her application in 
the future when the roles are exchanged and the applicant becomes the reviewer and the reviewer the 
applicant. 

One high level NIH official complained that NIH lacks effective sanctions to back up the 
proscription against revealing the contents of peer review reports and meetings. The offender can be 
removed from the committee and barred from future committees." Since committee membership is 
hard work, however, this is not necessarily a great penalty, even though committee membership does 
lend an element of prestige. One official suggested that NIH be empowered to levy fines against peer 
reviewers who violate their confidentiality promises. 

10. Financial Conflict of Interest 

Under NIH's conflict of interest regulations, a panelist's proposal may not be reviewed by his 
panel and he must leave the meeting before the panel considered any proposal from his or her 
institution. The Division of Research Grants attempts to send panelists' applications to closely 
related committees, but they receive no special treatment there, and they occasionally suffer from the 
second committee's lack of technical expertise in the relevant area. Panelists are apparently willing 
to accept this reduction in the chances that their proposals will be funded during the time that they 
serve on the panels because of the additional insight into the working of the grant process and the 
prestige that panel membership entails. Program Directors in the Institutes monitor the results of the 
peer review process to see if panelists have been unfairly evaluated in the committees to which their 
proposals are sent, and they occasionally make a case to the relevant Advisory Council and upper 
level Institute staff for funding such proposals by exception. 

Financial conflicts of interest are of special concern in programs, such as the General Clinical 
Research Centers and Biomedical Research Technology programs and all of the SBIR programs, that 
are designed to stimulate commercial development. There is an obvious financial conflict of interest 
when a scientist from a university sits on a peer review committee reviewing the application of a for- 
profit entity in which the reviewer has an economic interest. There is a less direct conflict of interest 
when a scientist from a competitor (or a university scientist with financial ties to a competitor) sits on 
the review committee for a grant proposal from the private sector. In the latter situation, the 
reviewer will no doubt become privy to commercially valuable information. If that information is 
conveyed to the competitor, it could receive an unjust commercial advantage. The guidelines do not 
explicitly address financial conflicts of interest that might arise by virtue of a panel member's 
financial stake in a private research company; nor do they address the possibility that commercially 
valuable information could be revealed to commercial competitors.** 

In programs with a small number of applicants and awardees, like the General Clinical Research 
Centers and Biomedical Research Technology programs and the SBIR programs, another form of 
subtle financial conflict of interest can enter the picture. In such programs study panel members are 
invariably drawn from institutions with existing funded projects. They know that to the extent that 
the projects from other institutions do not get funding, more money will be available for their own 



'*rhis unction was invoked on one occasion in NIH when a reviewer overheard another reviewer calling applicanu and telling them 
their priority scores on a pay phone. Telephone interview with Dr. Thor Fjellsiedt, supra note 25. 

**NIH in fact actively recruits scientists from industry to sit on SBIR review panels, despite the ohvious potential for appropriation of 
commercially valuable information. 
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institutions. In large programs with many applicants this incentive to downgrade other proposals is 
probably not very strong, but when only 10 or 15 major centers are funded in the country, "killing 
off' one center may substantially increase the probability that the remaining centers will received 
continued funding. 

11. Research Conflict of Interest 

NIH staffers attempt to fill the study sections with scientists of the very highest caliber. When a 
study section is composed of active scientists from a fairly narrow scientific field, it is always 
possible that a reviewer will use information obtained during the review process to advance his own 
research agenda. Alternatively, an overly aggressive reviewer may attempt to slow down the 
research of a competitor by giving its grant application a low ranking. One disgruntled researcher 
interviewed for this project related an incident in which both tactics were employed. After making a 
major discovery in research funded by the federal government, he met a prominent scientist at a 
convention and naively explained his theory to the scientist. The scientist complimented him on the 
quality of his work. When the young scientist submitted the project for competitive renewal after 
two years, the older scientist was highly critical of the project, and it was not renewed.'' Several 
years later the young researcher read a paper published by the older scientist "and he did exactly 
what we did and published it." Another researcher reported that after the wife of one of the most 
prominent researchers in his field sat on a study section evaluating one of his proposals, the 
prominent researcher suddenly began to direct his research in the direction indicated by the 
applicant's research proposal .'^ 

NIH staffers recognize the potential for "research conflict of interest" in which the rewards are 
not so much in coin as in prestige, but they report few instances in which it has been observed. By 
its very nature, of course, this nonfmancial conflict of interest is difficult to detect. NIH has no 
formal policy addressing research conflict of interest, and NIH staffers apparently do not address the 
problem unless an applicant requests that a particular scientist not serve as a review or his or her 
proposal. In the final analysis, the NIH professional staff relies on the honor of the participants in 
the review process. 

12. Peremptory Challenges 

Although NIH regulations do not give applicants the right to object to particular persons as 
reviewers for their applications, agency staffers often attempt informally to accommodate applicant 
concerns about potentially biased reviewers. This is especially true in the case of programs involving 
site visits where anonymity cannot be preserved. If an applicant complains in advance that a potential 
reviewer is likely to be biased because of past associations, research conflict of interest or even 
personal animosity, the staffer usually tries to avoid placing the potentially biased reviewer on the 
site review team. Even in programs using anonymous reviewers, if an applicant suspects that a 
potential reviewer will be biased against his proposal, the staff will often make an effort to see to it 
that the application does not go to a panel on which that reviewer sits. There are, of course, limits to 
which staffers are willing to insulate an application ft-om reviewers that the applicant deems biased; 
they will not, for example, allow a maverick to exclude every scientist in an entire field of research. 

13. Lobbying 

Instances of academic applicants' lobbying members of peer review panels and upper level staff 
are virtually unheard of. Lower level NIH staffers make a conscious effort to be available to 



'^The researcher who related this story is confident that he correctly ascertained the identity of the reviewer who nude the negative 
comments, because the jargon used in the criticism was the same arcane jar;gon thai the researcher had employed in the previous 
conversation. 

'^This researcher could not be certain that the change in direction was attributable to information gleaned by the competing 
researcher's wife in the study section, and he is willing to give the competitor the benefit of the doubt. Bui he noted that the potential for 
abuse is clearly present. 
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potential applicants to explain the system and to help guide deserving applications to the right study 
sections. They attend major scientific meetings and form friendships with scientists who are current 
or potential grantees, but they do not receive many arm twisting phone calls. In the relatively new 
area of SBIR grants, where the recipients are small companies rather than university research 
laboratories, there have been a few attempts to lobby the staff on the merits of particular proposals, 
but they are generally resisted as inappropriate.'' The contacts between applicants and lower level 
staff that do occur can have little impact on the outcome of the decisionmaking process, because the 
lower level staff have virtually no input into the initial priority scorings, which are done by peer 
reviewers in the study sections with minimal staff input. While the staff can have some influence at 
the margins in briefing the upper level staff prior to Advisory Council meetings, major departures 
from the rankings of the peer review panels are very rare and difficult to justify. 

Occasionally, an applicant will directly contact an Advisory Council member to state his or her 
case. If an Advisory Council member can be persuaded to champion the cause of a rejected 
applicant, there is a reasonably good chance that the proposal will be funded. One NCI staffer 
suggested that to some extent the Advisory Councils operate as "buddy systems"; members of the 
Council are willing to speak up for their friends. NIH procedures do not expressly prohibit attempts 
by disappointed applicants to lobby members of the Councils to reverse the recommendations of the 
peer review sections. 

14. Political Pressure 

In NIH programs that administer a large number of relatively small grants to individual 
researchers, there are almost no reports of attempts to use the outside intervention of politically 
powerful actors, such as congressmen or other officials in the Administration, to pressure the agency 
staff to award grants to particular researchers. Even in the programs that award very large grants to 
companies and institutions, there are apparently very few attempts to use politically powerful friends 
to influence the outcome of the process. Most contacts by congressmen come in the familiar form of 
a written status inquiry that is easily answered by a polite letter explaining that the proposal did not 
have the "scientific quality" of the funded proposals and paraphrasing the comments on the pink 
sheet. The pink sheets themselves and other review information are not made available to 
congresspersons, unless they are requested in connection with an official congressional investigation 
of NIH. Some NIH staffers and researchers ventured that applicants might be reluctant to rely upon 
this kind of influence-peddling out of fear that the strategy might backfire by alienating the staff and 
members of future study sections. In any event, none of the NIH staffers interviewed for this Report 
expressed any concern that outside political pressure was a significant factor in the funding process. 
None of them could remember a single instance in which a funding decision was changed because of 
outside political pressure. 

In more recent years, congressmen and senators who are concerned about channeling funding for 
large projects into their districts have taken the more direct approach of putting the funding into 
legislation, often by way of a rider to a different bill or a special clause in an appropriations bill. 
This technique for avoiding peer review, which has generated much controversy in the scientific 
community, will be discussed in more detail in Part 3. 

15. Efficacy of Appeal Procedures 

Although NIH has a comparatively extensive appeals process prior to the final funding decision, 
the agency staff typically discourages rejected applicants from appealing funding decisions. Instead, 
applicants are urged to study their "pink sheets" and submit a revised application during the next 
round of applications. While the Council and the panels are theoretically willing to revisit grant 
applications in cases in which legitimate concerns are raised, such "rereviews" occur only very 



"One NIH slafTcr related the experience of the Vice President of a small oil company who was amazed thai his company received a 
SBIR grant, even though the suff never accepted any of his frequent luncheon inviutions 
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rarely. As a practical matter, a rewrite and resubmittal can result in the effective reconsideration of 
the proposal in not much more time than it would take the appeals process to run its course with a 
proposal that by the end of that process might be outdated. Once the final funding decision is made 
by the agency, no appeals are allowed. Since the agency's regulations do not specify a vehicle for 
judicial review, the disappointed applicant is apparently left to his or her remedies under the 
Administrative Procedure Act. 
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III. PEER REVIEW IN THE NATIONAL SCIENCE FOUNDATION 

The National Science Foundation (NSF) was created by Congress in 1950 with the broad goal of 
promoting and advancing science in the United States.* Whereas the National Institutes of Health 
(NIH) focuses almost exclusively on research related to human health, NSF supports research across 
many broad areas of the physical, natural and social sciences. Because NSF does not have its own 
research facilities, nearly sdl NSF-supported research is conducted in university laboratories and 
laboratories administered by university consortia. NSF also sponsors a relatively small amount of 
research in laboratories run by other govemment agencies (e.g. the Argonne National Laboratory and 
the Los Alamos Scientific Laboratory) and, more recently, at for profit laboratories. NSF distributes 
more than $2 billion annually to more than 17,000 grantees." During the 1980s, NSF experienced a 
40% increase in the number of applications for research grants, while its funding remained relatively 
flat."" Fewer than 45 percent of the principal investigators who apply to NSF for funding are 
successful . '<" 

The National Science Board (NSB), composed of 25 persons (usually prominent scientists, 
engineers, and occasionally public figures with an interest in science) appointed for six-year terms by 
the President with the advice and consent of the Senate, is the primary policymaking organ of the 
institution. The NSB must approve most new programs and nearly all grants or contracts totaling 
more than $6 million or involving annual expenditures of more than $1.5 million."^ 

A. Discretionary Grant Programs 

An application for NSF research can be made to a Program in any one of its eight broad 
Directorates."" Most grant applications come from educational institutions or consortia of 
educational institutions, a few come from for profit companies (mostly small businesses), and a 
smattering come from individuals. In recent years NSF has strongly encouraged small businesses to 
apply for grants under its Small Business Innovation Research (SBIR) program, and it also 
encourages university-industry collaborative studies.'** Project grant funds may be used for paying 
costs necessary to conduct research or studies such as salaries and wages, permanent equipment, 
expendable equipment and supplies, travel, publication costs, and other direct and indirect costs. 
NSF has recently established a procedure under which a Program can award up to 5 percent of its 
budget in small grants of not more than $50,000 on an expedited basis without peer review.'*" This 
report will focus on two of the many NSF programs~the Program on Biotic Systems and Resources 
in the Directorate of Biological Sciences and the Program on Science and Technology Centers in the 
Directorate of Major Initiatives and Other Activities. 



'^Thc National Science Foundation was established by the National Science Foundation Act of 1950, 42 U.S.C. §§1861-187S. 
Congress provided additional authority in the Science and Engineering Equal Opportunities Act, 42 U.S.C. §§1885-]885d, and tilJe I of 
the Education for Econonuc Security Act, 20 U.S.C. §§391 1-3922. 

•^National Science Foundation. Guide to Programs FY 1991 (1991) [hereinafter cited as 1991 NSF Program Guide). 

""^Merit Review Task Force, National Science Foundation, Report of the Merit Review Task Force (August 23, 1990), at 6. 

""l>erining a successftil principal investigator as one who has received at least one award during a 3-year period, the success rate 
dropped from 45% to 42% from the 1980-82 to the 1987-89 period. Merit Review Task Force, supra note 100, at 12. 

""See 1991 NSF Program Guide, supra note 99, at vii. 

'^'^The Directorates of NSF arc: Biological Sciences; Computer and Information Science and Engineering; Education and Human 
Resources; Engineering; Ceosciences; Mathematical and Physical Sciences; Scientific, Technological, and International Affairs; and Major 
Initiatives and Other Activities. 1991 NSF Program Guide, supra note 99, at v-vi (1991). 

'*^199i NSF Program Guide, supra note 99, at vii. See also See National Science Foundation, Small Business Innovation Research: 
Program Soliciution (1990). According to NSF's General Counsel, the idea for Small Business Innovation Research awards originated at 
NSF. Interview with Mr. Charles Herz, General Counsel, and Mr. Matthew Powell, attorney. National Science Foundation. March 13, 
1992, Washington, D.C. [hereinafter cited as Herz & Powell Interview]. 

"*^See David Bjerklie, Fast-track Grants: NaliorutI Science Foundation's Small Grants for Exploratory Research, 93 Technology 
Review 19 (1990). 
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1. Biotic Systems and Resources 

The Biotic Systems and Resources Division, which is part of the Biological, Behavioral and 
Social Sciences Directorate, funds basic research in five different fields of traditional systematics and 
ecology.'"* Within that Division, the Biological Research Resources Program provides support for 
the curating, refurbishment, and computerization of systematic research collections of preserved 
plants, animals and fossils with the goal of conserving essential scientific resources at the national 
and international level and making those resources available for scientific study. Research in the area 
of Systematic Biology explores the identities, relationships and distributions of plants, animals, and 
microorganisms, and through the study of living organisms and fossils, it documents changes in 
biotic diversity through the earth's history. NSF funded research in ecology provides a framework of 
concepts on the ecology of land and inland waters and studies outside influences on the distribution 
and abundance of animals and plant communities. The Ecosystem Studies Program funds field, 
laboratory, and mathematical modeling research on ecosystems. The program on Population Biology 
and Physiological Ecology funds, inter alia, studies on the mechanisms by which traits in individuals 
are translated into characteristics of populations.'*" 

The Division has a budget of around $65 million.'" The size of the grants ranges from very 
small grants of only $5,000 to very large grants of $1,000,000. Most of the awards are for less than 
$100,000, and the average grant is about $70,000.'^ Approximately 15 percent of all applications 
are fiilly funded; 10 percent are partially funded; and 15% not funded due to lack of funds. The 
remaining applications are declined for lack of scientific merit or program relevance."" "Proven 
producers" have a 35-40 percent chance of receiving funding for any given application.'" 

2. Science and Technology Research Centers 

The Program for funding science and technology research centers is a relatively new program 
aimed at providing "mechanisms to exploit opportunities in science and technology where the 
complexity of the research problems or the resources needed to solve them require the advantages of 
scale, duration, facilities, or collaborative relationships that can best be provided by campus-based 
research centers.""^ The goal is to "help maintain U.S. preeminence in science and technology by 
funding research activities that call for cooperation among a group of scientists, engineers, and their 
students.""^ The monies, which come in large five-year grants, are used "to fund comprehensive 
programs where specific problems need center-like funding to ensure world-class quality and 
competitiveness.""^ Although the program focuses primarily upon basic research, most centers 
address research topics in areas with easily identified technological applications. Examples include 
funding for centers on "superconductivity," "particle astrophysics," "parallel computation," 
"microbial ecology," and "polymeric adhesives and composites.""^ 

Since the program was established in 1989, it has gone through two funding cycles in which 
funds have increased to its current annual level of around $40 million."* In its first two 



"*CFDAal798. 

""1991 NSF Program Guide, supra note 99, at 3-4 (1991). 

'"^ConimenU of Mr. Garth Redfieid, Director, Environmental Sciences Division, Research and Evaluation Department, South Florida 
Water Management District, on an earlier draft of this report, August 14, 1991. 

»"CFDA at 799. 

'"telephone interview with Dr. Gatnh Redfteld, Former Assoc. Program Manager for Ecology Programs, National Science 
Foundation, October 17 and 24. 1990. 

"'CommenU of Mr. Garth Redfieid, supra note 108. 

"^FDAat808. 

"'1991 NSF Program Guide, supra note 99, at 94 (1991). 

"^1991 NSF Program Guide, supra note 99, at 94 (1991). 

"^CFDAal809. 

"*Herz & Powell Interview, supra note 104. 
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competitions, the program received about 470 applications and awarded a total of 25 grants ranging 
from $900,000 to $4,250,000 per year with an average of around $1,700,000."^ 

B. Criteria for Awarding Grants 

Grants for most of NSF's programs can be made to public and private colleges and universities, 
nonprofit, nonacademic research institutions, and private profit organizations. Science and 
Technology Research Center awards, however, may only be made to public and private academic 
institutions. The applicant institutions are expected to share in the support of NSF-funded projects, 
either directly or indirectly by providing facilities, equipment, maintenance, etc. 

General criteria for evaluating proposals include: (1) research performance competence- the 
capability of the investigator, the technical soundness of the proposed approach, and the adequacy of 
the institutional resources available; (2) intrinsic merit of the research-the likelihood that the 
research will lead to new discoveries or fundamental advances within its fields; (3) utility or 
relevance of the research— the likelihood that the research can contribute to the achievement of a goal 
that is extrinsic or in addition to that of the research field itself, and thereby serve as the basis for 
new or improved technology or assist in the solution of societal problems; and (4) effect of the 
research on the infrastructure of science and engineering-the potential of the proposed research to 
contribute to better understanding or improvement of the quality, distribution, or effectiveness of the 
Nation's scientific and engineering research, education, and manpower base."* 

C. Procedures for Awarding Grants 

The NSF peer review process does not rely as heavily upon panels as the NIH system, and it 
assigns considerably more discretion to the staff. For that reason, the role that peer review plays can 
vary from program to program within NSF.'" When an application is assigned to a Program Officer 
with jurisdiction over the relevant subject matter, he or she undertakes a preliminary assessment of 
the proposal's subject matter and attempts to identify a group of up to ten qualified peer reviewers 
from among those persons in the country with expertise in the subject matter.'^ Program Officers 
use several resources for identifying reviewers, "ranging from lists of reviewers suggested by the 
applicants or current peer reviewers themselves, to contacts made by NSF staff at professional 
meetings."'^' The most frequently relied upon source of reviewers is a computerized list compiled 
and maintained by NSF staff that contains the names of thousands of potential reviewers arranged 
according to areas of expertise.'^ Program officers attempt to select reviewers on the basis of their 
expertise, objectivity, open-mindedness, and (in the case of reviewers who may be assigned to 
panels) ability to work with others. 

Most peer reviews in NSF are "mail reviews" in which the Program Officer sends the ten or so 
proposed reviewers a copy of the application and a document setting out the review criteria. The 
letter asks the recipient to provide a written critique of the application and a rating based on the 
identified criteria. Since each mail reviewer receives only one of the applications in a given pool, the 
reviewer has no opportunity to compare the application that he or she evaluates with competing 
applications. Usually about five or six of the ten proposed reviewers respond to the request. 

About one-third of the applications also undergo a panel review. Some programs (e.g. biology) 
rely quite heavily upon panels, whereas others (e.g. chemical and physical sciences) use panels only 
rarely for especially complicated or expensive proposals. In a panel review, the assembled experts 
(usually eight-to-fifteen in number) read the mail reviews and attempt to evaluate all proposals in a 

"'CFDAat 809. 

"*1991 NSF Program Guide, supra nole 99, at ix; CFDA at 799. 

'"Commenu of Mr. Garth RedTield, supra note 108. 

'^"^In 1985 NSF asked almost 60,000 persons to serve as external peer reviewers. 1987 GAG Report, supra note 45. 

'-'1987 GAG Report, supra note 45, at 14. 

'Comments of Mr. Garth Redfield, supra note 108. 
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given pool.'^ Programs making very large awards, like the Science and Technology Research 
Centers program, also require site visits by peer panels. Like NIH, the panels usually allocate the 
work load by assigning each application to a subcommittee of two or three persons who are primarily 
responsible for the evaluating that application. The panelists are usually invited to rank the 
proposals. 

After the individual and/or panel reviews have been completed, the Program Officer evaluates 
and ranks the proposals, taking into account the peer evaluations of the technical merit of the 
proposals and other factors of a less scientific nature such as "infrastructure"'^ and " equity" '^^ 
concerns. When proposals have been submitted to a panel, the Program Officer must provide a 
written justification for any deviations from the panel's recommendations. In practice there is a high 
degree of correlation between panel recommendations and funding decisions.'^ The Program 
Officer's decisions are in turn reviewed by section heads and in some cases the head of the 
Directorate before final funding decisions are made. At the upper echelons additional considerations 
are factored into the decision, including long term political concerns, technological innovation, and 
potential for practical application in the private sector, geographical equity in the distribution of 
ftinds, and (to a somewhat lesser extent than at the Program Officer level) overall scientific merit. 
On very rare occasions, upper level decisionmakers will appoint advisory committees to provide 
input into the decision. Decisions to award grants of more than $1.5 million per year must be 
reviewed by the National Science Board, the ultimate decisionmaker at NSF.'=^ The entire process 
normally takes about five to six months for individual research grants'^ and from 10-12 months for 
major institutional grants such as grants for centers under the Science and Technology Centers 
Program.' 29 

NSF has prescribed detailed conflict of interest regulations for persons employed by NSF, 
including members of the National Science Board. These regulations prevent present and past 
employees from representing anyone in dealing with any federal official in any matter in which they 
were involved at NSF."° They also prohibit any NSF employee from becoming personally involved 
in the handling of any proposal or other matter in which he, a member of his immediate family, or an 
organization of which he is a part or may become a part has a financial conflict of interest. ''' Finally, 
NSF's regulations provide that persons employed by NSF who have access to information not 
generally available to the public may not use that information for their private benefit or for the 
private benefit of others."^ A member of the National Science Board may not participate in 



'About one-third of all proposals to the NSF are reviewed by mail reviewers alone. Another third are reviewed exclusively by 
panels of reviewers who gather, usually in Washington, to discuss their advice as well as deliver it. The remaining third are reviewed first 
by mail reviewers expert in the particular field, then by panels, usually with more diverse expertise, who help the NSF decide among 
proposals form multiple fields or subfields. Letter to Eric R. Glitzensuin from Charles H. Herz, dated March 12, 1990, at 2-3. 

These numbers can vary from Program to Program. For example, in the Biotic Systems and Resources Program, approximately 95 
percent of grant applications are reviewed by nuil. Telephone interview with Mr. Victor Westbrook, Grants Officer for Biological, 
Behavioral, and Social Sciences, National Science Foundation, November 2, 1990. 

'^According to one Program Officer interviewed for this Report, if a solid and consistent performer may lose his or her laboratory as 
a result of a decision not to fund an uninspiring application, the program officer may give it a somewhat higher priority to 'save* the 
laboratory as an infrastructure resource. Telephone interview with Dr. Garth Redfield, supra note 1 10. 

'^Program officers must pay attention to whether minorities, women, and young investigators are fairly represented and to the 
geographic distribution of the awards. Herz & Powell Interview, supra note 104. 

'Stephen Cole, Jonathan R. Cole and Gary A. Simon, Chance and Consensus in Peer Review, 214 Science 881 (1981). 

'^1987 GAG Report, supra note 45, at 9-1 1 . 

'^CFDA at 799. Herz & Powell Interview, supra note 104. 

'^CFDA at 808. 

'*45C.F.R. §684.14. 

"'45C.F.R. §684.16. 

"^45 C.F.R. §§684.1 1(h), 684.17(a). 
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deliberations and votes that would affect the member's own interests or those of a close relative, or 
an institution with the member or close relative has any of several designated affiliations.'" 

A separate set of regulations is directly applicable to outside peer reviewers. All peer reviewers 
are asked by the relevant Program Officer to reveal any possible conflicts of interest that the reviewer 
may have."^ The regulations define "potentially biasing affiliation or relationship" by reference to 
several kinds of relationships. "Affiliations with an applicant institution" include current employment 
or being under consideration for employment with the relevant institution, holding an office or 
membership on the governing board of the applicant institution, ownership of the institution's stock, 
current enrollment as a student in the institution (but only with respect to applications from the 
student's department), and receipt of an honorarium or award within the last twelve months. 
Relationships with investigators that have a personal interest in the grant include marriage, business 
partnership, past or present association as a thesis advisor or thesis student, and collaboration on a 
project within the last 48 months. Other disqualifying relationships include immediate family or 
relatives living within the same immediate household and "Ia]ny other relationship, such as close 
personal friendship, that [the reviewer] think[s] might tend to affect [his or her] judgments or be seen 
as doing so by a reasonable person familiar with the relationship."'" All such potentially biasing 
affiliations or relationships must be revealed to the Program Officers."* Most of these relationships 
are "automatically disqualifying," but some are only "partially disqualifying." 

The unique multi-tier peer review process that NSF has developed for awarding grants in the 
Science and Technology Research Centers Program departs from the typically applied model 
described above, because the awards in that Program are very large and last for at least five years. 
The review begins with the appointment of an initial multi-disciplinary 15 member "external peer 
review committee" to advise the staff during the entire funding cycle. After receiving all of the 
applications, the agency staff divides them into specific "buckets" (categories) according to discipline. 
Each bucket is distributed to a separate panel of experts assembled for the purpose of reviewing the 
applications and the outside reviews of the applications. Each panel is staffed by several Program 
Directors from the programs that deal with the subject matter of the applications. The Program 
Directors identify 8-10 outside "mail reviewers" for each proposal and arrange in advance by 
telephone for the reviews. This assures a large response rate of about 90 percent. One of the 
outsider reviewers is also a panel member. 

After the mail reviews are complete, they are sent to panel members. Two or three panel 
members are assigned the responsibility for detailed review of each application. The panels then 
assemble for two days to discuss the proposals. Since the agency cannot realistically attempt more 
than 30 site visits,'" each panel is told to recommend no more than three or four proposals for further 
consideration. This usually represents only about 10-15 percent of the applications considered by the 
panel. The staff may add an additional 6-8 proposals to the pool recommended by the panels to make 
a total of thirty. This latter step is an attempt to provide some flexibility to allow for any unevenness 
in the quality of the proposals across the six panels. It also gives the staff some discretion to 
overrule panel decisions. The applications are then forwarded to the original 15 member external 
peer review committee for discussion and a determination of which facilities will receive site visits. 

Since all applicants that do not receive site visits are eliminated from the competition, the agency 
at this point informs all applicants of the results of the process. Applicants receive verbatim copies 
of all mail reviews (with reviewers' identities redacted) and copies of the summaries of the panel 
meetings relevant to their proposals. The panel summaries are intended to give the applicants some 



'"45 C.F.R. §684.21. The proscribed aflilialions include current employmenl, formal arrangemenis for future employment, 
employment as an adjunct professor, etc. 

'^45 C.F.R. §68 1.25(a). 

'"45 C.F.R. §681.21. 

"*45 C.F.R. §681.25. 

"^This conclusion was based on the experience of the first Science and Technology Center competiiion. Herz & Powell Interview, 
supra note 104. 



Peer Review in Discretionary Grantmaking 29 



idea of how their proposals fared in relation to the others that the pane! considered. Typically, the 
panel summary is drafted by the panel member who presented the proposal to the panel, and it is 
circulated to the remaining panel members for suggested corrections or additions. The 30 or so 
applicants that are chosen for site visits receive the above information and a list of questions to be 
answered in anticipation of the site visit. They are also invited to comment on the mail reviews and 
panel summaries. Finally, the 30 remaining applicants are asked to provide two-page updates of their 
proposals to identify any changes that have occurred during the 6-8 months since the original 
proposals were submitted. Applicants may also make changes in priorities and in their budgets at 
this point. 

Site visits are conducted by teams of around eight members consisting of one or two members of 
the peer review panel that reviewed the original proposals, one or two staff members, one member of 
the 15 member external review committee, and other scientists with expertise in the relevant area. 
The site visit team receives all of the information that is in the file about the proposal, including the 
institution's response to any questions posed by the external peer review committee. After meeting 
with officials and researchers at the applicant institution for two days, the site visit team prepares a 5- 
6 page report discussing the strengths and weaknesses of the proposal with respect to the quality of 
the researchers, depth of institutional support and overall educational climate at the institution. The 
site visit team must answer specific questions posed by the external peer review committee and 
identify the strengths and weaknesses of the application, but it is not supposed to recommend whether 
the proposal should or should not be funded. Because the agency staff has some fear that site visit 
teams will become "cheerleaders" for the sites that they visit, they request that the reports be as 
dispassionate and objective as possible. 

The site visit reports are given to the external peer review committee, which has by now been 
expanded from fifteen to twenty-five members to lend additional expertise to the consideration. Each 
panel member is given a large binder with an abstract, executive summary, budget and all reviews of 
each of the 30 proposals, but the work of providing intensive reviews of individual proposals is 
assigned to subcommittees of three reviewers. The external peer review panel is divided into two 
groups for the detailed consideration of the proposals. The reason for this ftirther subdivision is the 
practical impossibility of ensuring that no member of the 25-member committee participates in the 
evaluation of an application from his or her own institution. Since it is virtually certain that one or 
more members will be from one or more institutions with proposals before the committee, the staff 
ensures that no member is on the half of the committee that undertakes the first assessment of the 
proposal from his or her institution, and that person also leaves the room while the second half of the 
committee considers the first evaluations. Each group is asked to rank all of the proposals and to 
choose the top 7-10. On the second day of the meeting, the groups exchange rooms and each reviews 
the work of the previous group to see whether it agrees with the prior group's rankings. On the third 
day, the committee attempts to arrive at a consensus on about 17-20 proposals that deserve funding. 
The Director of the Science and Technology Center Directorate then recommends 10-15 from among 
the proposals recommended by the external peer review committee for presentation to the National 
Science Board, the ultimate policymaking entity in NSF. 

One NSF staffer from another program suggested that the peer review process in place at the 
Science and Technology Centers Program is "the best in the world. "'^ There can be little doubt that 
it is one of the most thorough and complex peer review systems in the world. The redundant layers 
of peer review are justified by the large sums of money that are at stake and the temptation that 
would otherwise exist to fund centers in a biased fashion on basis of criteria other than scientific 
merit. 



"^Telephone interview with Dr. Garth Rcdfield, supra note 1 10 
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D. The Process in Operation 

The procedures in NSF for making information available to applicants and the public and for 
allowing rebuttals and challenges to nonfunding decisions have been evolving over the last several 
years from a relatively closed and unresponsive system to a comparatively open one. This evolution 
was nudged along by a petition from a researcher who (with the support of a Ralph Nader-affiliated 
public interest organization called Public Citizen) had the persistence to prove that the system had 
wronged him and the courage to demand that the agency ensure that what happened to him would not 
happen to future researchers."' 

1. The Kalb Litigation 

In the late 1970s, Jon Kalb applied for an NSF grant to continue his anthropological research in 
Ethiopia.'* The research, for which Kalb held an exclusive concession from the Ethiopian 
government, was the subject of three separate proposals by principal investigators from Southern 
Methodist University, New York University and Harvard University. He later learned that the NSF 
staff had informed the peer reviewers for all three of these proposals of rumors that he was associated 
with the Central Intelligence Agency (CIA) and that his scientific efforts were merely a cover for 
covert aaivities in that country. Kalb alleged that one source of the rumors was one of the reviewers 
assigned to his grant application who in turn learned of them from the head of a rival research group. 
In fact, one of the reviewers of the Harvard proposal wrote in his review the following: "The 
qualifications of the senior personnel are beyond question and this reviewer is fully in sympathy with 
Mr. Kalb's efforts and energy in getting the (project] together. However, his possible involvement 
with the CIA must, until this has been cleared up, remain a matter of concern for any scientists."'*' 
The scientist who wrote this review had hoped to become a collaborator at Kalb's site, but Kalb 
declined his overture just a few weeks before the review was written. It is reasonably clear from 
after-the-fact attempts to reconstruct the meetings of the peer review panels that Kalb's alleged 
involvement with the CIA was a prominent consideration in the panels' deliberations, even though it 
was never mentioned in the written minutes of those meetings.'*^ After all three applications were 
rejected and after Kalb and his family were expelled from Ethiopia because of the same rumors, the 
reviewer who brought the CIA rumors to the attention of one of the panels applied for and received 
an amendment to an existing grant allowing him to use NSF monies to take over the concession that 
the Ethiopian government had previously awarded to Kalb.'*' 

Kalb attempted to challenge the decision not to fund his project, but he was blocked at every 
turn by the secrecy that NSF demanded for the peer review process and by the staff's general lack of 
responsiveness. After ten years of challenges, during which he attempted to demonstrate that the 
rumors were false and that they had affected the decision on his proposal, Kalb finally sued the 
agency. The lawsuit resulted in a settlement under which NSF issued a formal apology to Kalb and 
paid his attorneys' fees of approximately $20,000.'*' 



'^^SF attorneys dispute the conclusions thai any significanl change was brought about as a result of the Nader-sponsored Kalb 
rulemaking petition. Noting that the Kalb petition was unique in the history of NSF, they maintain thai the agency had already adopted 
and had been implementing nearly all of the changes that Kalb demanded in his petition. They contend that in many ways (e.g. the ability 
of the applicant to read the actual written reviews of the reviewers) the NSF system has always been more open than the NIH system. 
Many NSF ttafTers believed that the Kalb petition was 'making a mountain out of a molehill,* because nearly all of the information that 
Kalb was requesting would have been available to him without his reliance on the Privacy Act. Herz & Powell Interview, supra note 104. 

'*^e proposal was subnutted jointly by Kalb and researchers from Southern Methodist University and New York University. The 
story of the Kalb petition is related in considerably greater detail in Robert Bell, Impure Science: Fraud. Compronusc, and Political 
Influence in Scientific Research, ch. 1. 

'*'ld.at21. 

'*2id. at 16-20. 

'*Md. at 22-23. 

'**Kalb v. CIA, CrV. No. AE6-3557 (D.D.C. Dec. 8, 1987). Sec NSF Admits Spreading Spy Rumor; Agency Apologizes To Grant 
Applicant, Washington Post, December 4, 1987, A25. 
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Not long after the settlement, Kalb and Public Citizen organization filed a petition with NSF 
asking the agency to write new procedures, pursuant to the Privacy Act'*^ and the Federal Advisory 
Committee Act,'** for providing applicants with access to information concerning their applications 
and for giving them a realistic opportunity to rebut any false allegations made during the review 
process. Unlike NIH, NSF had not officially maintained a "system of records" capable of retrieving 
information about an individual by name or personal identification number, and it therefore had not 
subjected its peer review proceedings to the Privacy Act. Kalb's attorneys argued that the agency 
had nevertheless maintained a de facto "system of records" in violation of the Privacy Act. 

The petition demanded that NSF correct four alleged basic flaws in its peer review system: 

(1) applicants are being deprived the right to gain access to, and the opportunity 
to amend, vital information considered in the peer review process; 

(2) applicants are not afforded advance notice of, and an opportunity to rebut 
non-scientific derogatory information prior to final decision-making; 

(3) applicants do not have the opportunity to prevent conflicts of interest by those 
involved in the review and evaluation process; and 

(4) the appeals process is woefully inadequate.'*' 

Acknowledging that the peer review system must be kept confidential from grant applicants to 
ensure frank and candid options and that peer reviewers must therefore remain anonymous, the 
petition maintained that NSF had gone far beyond what was reasonably necessary to ensure the 
confidentiality of the review process. Kalb demanded that NSF "amend its procedures to ensure that 
grant applicants have sufficient access to and the opportunity to amend, records regarding the 
consideration of the grant proposals."'^ 

The petition first urged NSF to acknowledge its Privacy Act obligations to make information in 
review files available to applicants along the lines of the NIH model and to inform applicants of their 
Privacy Act rights. Second, the petition demanded that applicants be given access to the contents of 
all outside reviews prior to any final decisions on their applications. Third, it asked NSF to comply 
with its Federal Advisory Committee Act obligation to make complete minutes, rather than staff 
summaries, of peer review panel meetings available to applicants. Fourth, the petition insisted that 
NSF adopt special procedures to give applicants an opportunity to learn about and rebut derogatory, 
harmful or non-scientific information and allegations of misconduct before the final decision 
regarding a grant proposal. Fifth, the petition requested NSF to adopt a procedure for allowing 
applicants to play a role in guarding against research conflicts of interest.'** Acknowledging that 

NSF attorneys maintain that the payment of attorneys' fees was solely for the purpose of eliminating the 'nuisance value' of the 
lawsuit and in no way constituted an admission that the agency had wronged Kalb in any way. In particular, NSF attorneys maintain that 
the rumors did not affect the outcome of the agency's treatment of Kalb's proposal. Herz & Powell Interview, supra note 104. 

'^5 U.S.C. §S52b. The Privacy Act requires federal agencies to protect personal infomution in agency files from unauthorized 
disclosure, to publish publish descriptions of the existence and nature of the records conuining personal information about people, and to 
give individuals access to review an copy information about themselves and to demand that the agency correct any information that is not 
accurate, relevant, complete or timely. The Act, however, only applies to 'systenu of records* from which records are retrieved by the 
name of an individual or other personal identifier. This latter qualification has proven controversial an difficult to interpret. 

'^S U.S.C, App. 2. The Federal Advisory Committee Act requires federal agencies that rely upon recommendations of advisory 
committees to charter those committees. The charter must set out the committee's objectives, duties, number and frequency of meetings, 
and termination date. The agency must prepare minutes for advisory conrunittee meetings and make those minutes available to the public, 
subject to the exemptions in the Freedom of Information Act. 

'^'Petition for Rulemaking by Public Citizen and Jon Ervin Kalb, July 13, 1989, at 4. 

'«ld. 

'^^The petitioners pointed out that: 'While NSF's rules recognize the need to guard against traditional, financial, institutional, and 
personal conflicts of interest, they say nothing about the kinds of conflicts thai may be of even greater concern to scientists— direct conflicts 
of interest regarding the specific research covered by a particular proposal.* Id. at 26. The petitioners noted thai *|ilhere is obviously a 
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allowing applicants to play a role in identifying potential conflicts of interest might threaten the 
confidentiality of the reviewers' identities, the petition suggested that NSF could make available a list 
of potential reviewer to applicants in advance and allow applicants to object to any listed scientists 
that might have a research conflict of interest. Kalb hoped that such a procedure would help avoid 
conflicts in which a researcher could potentially affect the prospect of funding for a major 
competitor. Finally, the petitioners asked NSF to amend its appeals process to make it more 
accessible to rejected applicants.'* Once again, the petitioners suggested the NIH appeals process as 
a model. '^' 

On March 12, 1990, NSF responded to the petition. Rejeaing most of the petition's allegations, 
the agency noted that it had already adopted most of the petitioners' suggestions.'^ Without 
conceding that it had maintained a de facto "system of records" the agency agreed to continue making 
all reviews, notes of telephone conversations with reviewers, and sunmiaries of the contents of any 
panel meetings available to applicants after the agency's final decision. Only the names of reviewers 
and the contents of competing proposals would be redacted. Written reviews and summaries of panel 
meetings would be mailed automatically to applicants. NSF further agreed to make greater efforts to 
inform applicants of their Privacy Act rights. 

The agency rejected the proposal that it establish an additional process for allowing applicants to 
rebut nonscientific statements and innuendo prior to making final awards decisions. Noting that it 
had already adopted a policy prohibiting the consideration of reviews demonstrating bias or 
containing personal attacks, NSF determined that it would be too burdensome to allow a formal 
rebuttal to all applicants who desired to challenge the process.'" The agency raised the difficulty of 
distinguishing rebuttals based upon alleged bias from rebuttals attacking the scientific merits of the 
reviews and rankings. NSF noted that declined applicants could submit petitions for reconsideration 
after-the-fact if they believed that they had been the objects of animus. 

On the question of research conflict of interest, NSF agreed to provide computerized rosters of 
all potential reviewers to applicants, to invite applicants to suggest the names of persons who might 
be biased against their proposals, and to give this invitation prominence by making it part of the 
agency's acknowledgment letters. Although the agency made no commitments to follow the principal 
investigators' suggestions, it anticipated that most would be honored.'** It declined to make public 
the names of potential reviewers that it decided to disqualify. 



great poteniul for bias if a scienlitt that is asked to review a grant application is already conducting the sante or very similar research to 
that being proposed by the grant applicant, particularly if the reviewer believes that the proposed research nuy, if funded, somehow 
preempt or hinder his own work.' Id. 

'**^e petitioners noted that NSF's office of the Deputy Director had recently concluded that only a total of five proposals were 
successfully appealed during the five-year period preceding 1985. Id. at 32. 

'*'The petitioners asked NSF to adopt rules amending iu appeals systems to provide that (1) principal investigators have an 
opportunity to examine and seek amendment of all non-exempt portions of records considered or generated by NSF in the course of 
denying a grant application; and (2) grounds for appealing a denial of a grant include a demonstration thai *(a) any records relied on in the 
decision are incomplete, inaccurate, untimely, or irrelevant; (b) a conflict of interest or bias influenced the adverse decision; and (c) there 
was subsuntive or procedural error in the process of peer review, including violations of the Privacy Act, the Federal Advisory Comnuttee 
Act. or other applicable federal sututes or any NSF regulations or guidelines.* Id. at 36. 

"^Letter to Eric R. Glitzenstein from Charles H. Herz, supra rKHe 123. For example, with the exception of telephone notes, all of the 
listed procedures had been followed in the Directorate of Biological Sciences for almost a decade prior to the Kalb petition. Conunenu of 
Mr. Garth Redfield, supra nou 108. 

"^The committee explained: 

fW]e concluded that the costs of such a formal and automatic process would outweigh the benefiu. Among the cosu 
would be weeks or nwnths of delay in the review of every proposal, thousands of hours of efTon by principal 
investigators, similar demands on the time and energy of already stressed NSF program surr(with consequent sacrifice of 
other services to the scientific community and the public), and frustration all around when little changes as a result. 

Letter to Eric R. Glitzenstein from Charles H. Herz, supra note 123, at 9-10. 
'**Id. at 13. 
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With respect to appeals, NSF noted that its recently implemented Privacy Act procedures would 
provide all applicants an opportunity to correct any errors in their application jackets after-the-fact by 
way of asking for reconsideration. The agency agreed, however, to amend its regulations to clarify 
that motions reconsideration would not be restricted to procedural grounds, but could also address 
bias, conflict of interest, and the scientific merits of the decision.'" 

The NFS response to the Public Citizen petition describes a system that is very similar to the 
system in effect at NIH with the notable exception that NSF allows rebuttal only after the final 
rankings and funding decisions have been made. The discussion below, which focuses on the same 
issues that have arisen in the context of NIH peer review, suggests that the changes have not been 
cost-free, nor have they been entirely effective. Nevertheless, they go a considerable distance toward 
eliminating the preexisting potential for bias that gave rise to the Kalb controversy. 

2. Feedback and Rebuttal 

Even before the recent procedural changes, NSF staff wrote a letter to rejected applicants 
explaining in greater or lesser detail (depending on how close the application was to the margin) the 
reasons for the rejection. The letters encouraged applicants who were nearly funded to apply during 
the next funding cycle and offered suggestions for improvement. Applicants who were far from the 
margin were subtly encouraged not to try again. In the past, a rejected applicant was limited to 
telephoning the Program Officer or perhaps the upper level program director. These sometimes 
impassioned encounters could take their toll on the morale of the Program Officers, especially when 
they could not share the peer reviewers' reports with the rejected applicants. NSF's policy of making 
redacted versions of all reviews, notes of telephone conversations with reviewers, and summaries of 
the contents of any panel meetings available to applicants after the agency's final decision should help 
alleviate this situation. NSF applicants will now be have access to 6-8 individual reviews, as 
compared to the comments of two council members that are included in NIH "pink sheets." Rejected 
applicants are generally appreciative of the feedback that these informational requirements provide. 

The agency still does not provide a rejected applicant an opportunity for rebuttal. NIH provides 
a limited rebuttal opportunity by circulating the "pink sheets" containing reviewers' comments to 
applicants in sufficient time for the applicants to prepare written rebuttals for the relevant Advisory 
Council prior to its consideration of the application. NSF could implement an effective opportunity 
for rebuttal by allowing applicants to submit written rebuttals to their Program Officers prior to their 
ranking of the proposals for submission to the section and Directorate directors. NSF took the 
position in response to the Kalb petition that this would be too burdensome, but it is no more 
burdensome than the process that has already been effectively implemented at NIH. One former NSF 
program officer suggested that comparing NIH with NSF is inappropriate, because individual 
program officers in NSF have much higher workloads that their equivalents in NIH.'** While it is 
true that NSF program officers are overworked, sometimes handling as many as 200 actions per 
year,'" they are generally available for questions and requests for status reports from applicants and 
potential applicants. The additional burden of the few rejected applicants that are likely to take 
advantage of the rebuttal opportunity should not be excessive. If the additional burden is too great, 
the answer may lie in reducing the workload on the program officers, rather than running a system 
that is perceived to be unfair. The NIH experience suggests that rebuttals are extremely rare and 
therefore do not consume many staff resources. Indeed, NSF's implementation of a formal 



'**Id. at 15. This change was implemented in July 1990. National Science Foundation, Important Notice to Presidents of Colleges 
and Universities and Heads of Other National Science Foundation Grantee Organizations, Notice No. 109. July 17, 1990. 
'^^ommenls of Mr. Garth Redfield, supra note 108. 
'"CommcnU of Mr. Garth Redfield. Id. 
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reconsideration process in response to the Kalb petition did not generate the predicted "wave of 
reconsideration requests."'^ 

NSF also alluded to the difficulty of distinguishing rebuttals based upon alleged bias from 
rebuttals attacking the scientific merits of the reviews or ranking, a distinction that has not frustrated 
the rebuttal process at NIH. In any event, some additional burden may well be outweighed by the 
additional sense of fairness that should accompany a process that allows rejected applicants to cure 
clear factual errors. 

3. Favoritism 

The mail review system in place for most grant programs at NSF is susceptible to favoritism on 
the part of Program Officers. The Program Officers, who select the mail reviewers, are generally 
familiar with their outlooks and biases. By assiduously or unintentionally selecting reviewers who 
are likely to look favorably upon a particular researcher or kind of research, a Program Officer can 
subtly "stack the deck" in a way that increases the likelihood of favorable reviews and decreases the 
likelihood of unfavorable reviews. Two procedures in place at NSF help ensure against "stacking" 
the reviews. First, Division Directors monitor the peer review selections of the Program Officers 
serving under them, and they keep an eye out to ensure that selection is not biased."* Second, many 
Program Officers rotate from program to program every 1-2 years, thereby ensuring that any bias is 
not perpetuated.'" 

4. Old Boy Network 

Just as in NIH, an old boy network can dominate the NSF peer review process. One NSF 
Program Officer candidly observed: 

There really is a conservative bias. This is a closed community of people and they 
all know each other. It is real peer review and can be incestuous, making it hard for 
a new university to break into a field unless the peer reviewer will let them in. The 
peer reviewer has incentive not to because it would often be his/her own institution 
that would lose out because they are competitive with the applicant. ... It is really 
a closed club where only certain schools can get funded for the most part. 

This suggests that the old boy syndrome can sometimes be so strong, especially in very small 
programs that award a few very large grants, that the reviewers almost have a financial conflict of 
interest. On a broader level, a study sponsored by the Los Angeles Times under the direction of 
California Institute of Technology political science professor Bruce Cain determined that prior to 
1982, 20 academic research institutions received 41 percent of all federal research money allocated to 
universities. The remaining 59 percent was distributed among the 570 institutions.'*' 

The power of the old boy network is revealed by a rare public outcry against an NSF decision to 
fund a large research facility at the Florida State University instead of the Massachusetts Institute of 
Technology. Although the peer review panel seemed to favor the MIT proposal, the State of Florida 
and the local business community rallied around the Florida proposal with promises of additional 



'^Mcfnorandum to Members of the Nalioful Science Board from Walter E. Massey, Director re: Annual Report on the NSF Proposal 
Review System, April 24, 1991, Attachment I, at 3. See also Memorandum to Members of the National Science Board from Walter E. 
Massey. Director re: FY 1991 Annual Report on the NSF Proposal Review Sysum. March 1992. Atiachmcni I. ai 2. 

'^Commenu of Mr. Garth Redfield, supra note 108. 

'**lt is. of course, also possible to "stack" review panels, but it would probably be difficult for the Program Officer to protect the 
identities of panel members from public disclosure. In addition, the panelisu themselves can detect and report insUnces of apparently 
biased selection. Contments of Mr. Garth Redfield, supra note 108. 

'*'Frank Clifford, Worrisome Trend; Research Funds: Not so Scientific, Los Angeles Times. 1 1/27/87, Part One. pg. 1, col. 1. 
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financial support that were unmatched by the MIT proposal.'" When MIT officials and 
Massachusetts representatives complained that the peer review panel had indicated that the Florida 
group was not as highly qualified from a scientific standpoint, "° southern representatives responded 
that NSF panels were generally biased toward the Northeast and Pacific Coasts.'*^ NSF staffers are 
quick to point out that the Florida-MIT controversy is very unusual, because the agency nearly 
always goes along with the peer reviewers, even when the staff suspects that an exclusionary old 
boy's club is at work.'" They also maintain that the staff usually goes to great lengths to ensure 
against regional biases. The Florida award over the contrary advice of the peer review panel may in 
fact reflect that very concern.'** 

5. Halo Effect 

The extent to which the halo effect affects rankings in NSF depends upon the program. In some 
programs, proposals from highly regarded researchers are routinely funded even though they are 
sometimes of uneven quality. Other programs take pride in their refusal to look beyond the last 
funded grant in evaluating an individual's application. One Program Officer reported that when 
"famous people jot down notes and throw them at us, we'll turn them down and love doing it." As a 
consequence, research careers in programs that do not give much credit for past performance tend to 
end relatively quickly. Forty-five year-old researchers close down their laboratories after twenty 
years, because they cannot compete with twenty-five year-old "hot shots" who put in 20 hour days. 
The cycle repeats itself as the younger researchers acquire additional responsibilities and begin to 
bum out. 

6. Personal Animus 

Although NSF receives very few complaints of personal animus on the part of mail reviewers, 
they are not unheard of. One researcher interviewed for this report maintained that the mail review 
of one of her applications consisted largely of a personal attack on her. NSF's new procedures 
(which were generally followed in the past in most NSF programs) allow an applicant to suggest "a 
brief list of persons who should not be asked to review a particular proposal,"'*' and NSF generally 
honors such requests. Although the staffers and researchers interviewed for this report did not report 
any instances of animus on the part of NSF staff, the outside reviewer system would allow a Program 
Officer to "stack the deck" against a particular person or point of view, just as the deck can be 
stacked to favor an applicant. As previously discussed. Division Directors keep an eye out for this 
possibility as part of their general oversight functions. 



'^^Charies A. Radin, New Forces Drew Lab From MrT, Boston Globe, 9/21/90, Metro/Region p. 1. The Florida legislature 
appropriated $28 million for construction and a $5 million per year operating subsidy. MTT proposed only to renovate an existing 
laboratory that was located in an old bakery. Mrf made no special effort in the state legislature or in Washington, D.C. 

'"l>avid L. Chandler, Kennedy, Kerry Query Science Award, Boston Globe, 10/6/90, p. 48; John Wilford, MTT Sales Panel Choice 
of Rorida for Magnet Cenur, New York Times, 9/7/90, AI6, Col. 5; Williait) Booth, MfT Indignant Over Contract Award by NSF, 
Washington Post, 9/7/90, A 13. 

'*'*The Chancellor of the University of Florida System responded to the criticism of NSF's award as follows: "The South has been 
gening cheated. If you look and count the money, you'll see where the money goes. I've been around long enough to know that all the 
wisest people in the world do not live in California and the Northeast." 

Representative Michael Andrews of Texas stated: There is what we call 'brother-in-lawing' going on. Many of the boards continue 
to be filled with academicians from the Northeast and West, and we never have the votes to direct some of these dollars back to the Sun 
Belt.* Nfiftali, Bendavid, untitled. Slates News Service, 9/30/88. 

"telephone interview with Mr. Victor Weslbrook, supra note 123. 

'**Frank Press, the President of the National Academy of Sciences offers a different explanation for the Florida Suic award. He 
guggesu that it reflecu a fundamental shift in the reasons for funding scientific research at the national level from national defense to 
economic development. °[T]his shift from national security to economic growth guarantees that science funding becomes more a function 
of politicking than dispassionate peer review as stales decide that they're entitled to their fair share.' Michael Schrage, Blurring the Line 
Between Funding Science and Funding Economic Growth, Washington Post, 10/5/90, F3. 

'*'NSF Notice, supra note 155. 
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One researcher observed that it is not uncommon for mail reviewers to vary dramatically in their 
evaluations of the same proposal. For example, sometimes a single proposal will be assigned a rating 
of "1" from one reviewer and a rating of "5" from another. It is possible to interpret instances in 
which a single proposal receives rankings at both extreme ends of the ranking system as evidence of 
animus, but it is more likely evidence of inconsistency due to the inability to articulate and apply 
precise criteria for scientific merit.'** 

The multi-tier review process in place at the Science and Technology Research Centers Program 
very effectively shields proposals from animus, once the proposals have made the 30-application 
cutoff. The site visit teams are not asked to evaluate the proposal numerically, and they are pressed 
toward a consensus answer to each of the specific questions that the agency staff poses to them in 
writing. It would be very difficult for animus to infect the exceedingly diligent review that the 
proposals receive at the four-day meeting of the external peer review panel. The great care that the 
agency took to shield the process from favoritism also effectively shields it from animus. 

7. Mavericks 

Like NIH, NSF has struggled with how to respond to "mavericks" who submit proposals that 
are outside of the dominant paradigm. NSF peer reviewers have difficulty distinguishing the 
potentially brilliant paradigm-shifting prof)osal from the hare-brained idea. According to one NSF 
staffer, it is always easy to fund the principal investigator with an excellent reputation and a perfectly 
designed study on a topic that is not very interesting. An NSF-funded researcher who has also 
conducted many mail reviews and participated in site visits, observed that there is a definite 
"conservative" bias in the peer review system in favor of "trendy" approaches and against proposals 
that suggest new or innovative approaches to uncovering scientific truths. 

Especially in times of tight funding, peer reviewers are very reluctant to take a chance with an 
unproven investigator who offers up a potentially fascinating project with a high probability of 
failure. The agency is more likely to fund an unorthodox proposal from an established investigator. 
Mail reviewers tend to view out-of-the-ordinary proposals very critically. Some Program Officers 
therefore believe that they have a responsibility to keep an eye out for mavericks who deserve 
funding and to shield them from negative peer reviews. According to one Program Officer, peer 
review panels should be instructed to look kindly upon proposals that are innovative and interesting, 
even if they have a few technical problems. Given their very heavy work load, however. Program 
Officers do not have much extra time to monitor the peer review process for bias against innovative 
proposals. 

NSF's unique program for awarding small seed grants for novel proposals outside of the formal 
peer review charmels is an attractive vehicle for avoiding the tendency of peer review to ignore 
mavericks. A Program Director may set aside up to 5 percent of the Program's funds for these 
expedited awards of up to $50,000. Because it vests complete discretion in the Program Director, it 
can be used to fund pet projects of little scientific value, but the $50,000 cap ensures that not too 
much money is devoted to any single project before it is forced to prove itself in the formal peer 
review process. According to one Program Director, NSF is funding some "really neat stufT 
through this very limited program. 

8. Tunnel Vision 

Sometimes peer reviewers in a narrow scientific field fail to see the relevance of proposals to the 
overall scientific enterprise or to the general welfare. For example, one researcher who applied to 
both NSF and EPA for grants reported: 

The success of a proposal often depends on the reviewers' perceptions of what 
constitutes technical merit. In 1982, my proposal received a very high rating from 



'*WhiIc this should provide some comfort lo those who fear that the system might be irrationally biased against them, it does not 
speak especially well of peer review as a vehicle for choosing projects with the greatest potential for success. 
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the EPA review panel. The very same proposal was not funded at NSF, despite 
generally good reviews. One reviewer in the biological program didn't like it, 
possibly because it had an engineering perspective. This negative vote was the kiss 
of death.'*' 

While it is fortunate that another funding agency was available to support the project, NSF's 
review was apparently affected by one reviewer's tunnel vision. 

9. Applicant Anonymity 

NSF makes no attempts to shield the identity of the applicant from the reviewers. One 
advantage of letting the reviewers know the identities of the applicants is that it greatly facilitates the 
ability of the reviewers to uncover plagiarism or stolen ideas on the part of applicants. On two or 
three occasions instances of plagiarism have been detected in this way in the Biotic Systems and 
Resources Program. These cases were referred to NSF's Inspector General.'"^ 

10. Reviewer Anonymity 

Some observers have argued that NSF'S reforms should go farther to reveal the identities of peer 
reviewers in addition to the contents of their reviews.'" Although this issue is fairly debatable, the 
vast majority of scientists involved in the peer review process oppose it on the convincing ground 
that candor would suffer dramatically if this reform were introduced. One NSF-funded researcher 
commented that excessive curiosity about who is on one's review panel is a "sign of paranoia," and 
predicted that revealing the identity of reviewers could be "debilitating" to the system. 

Public Citizen's less radical suggestion that applicants be allowed to review lists of potential 
reviewers of panelists and to raise objections prior to the consideration of his or her proposal should 
go a long way toward reducing the potential for research conflict of interest. The proposal would be 
far more usefjl, however, if the agency made an effort to provide more narrowly focused lists of 
potential reviewers to applicants who requested them. 

11. Financial Conflict of Interest 

There appears to be less concern among NSF staff and NSF-funded researchers about financial 
conflict of interest than in NIH. The biotechnology boom hit NIH earlier than NSF, and not as many 
NSF-funded researchers have private positions or consultantships with private companies. Concern 
in NSF will no doubt grow as university-industry ties spread into other areas of scientific research 
such as computer sciences. 

12. Research Conflict of Interest 

The Kalb story relates how a researcher with an interest in a limited resource can use his or her 
position as a reviewer of a competitor's grant application to enhance his or her own prospects for 
success. The potential for this sort of research conflict of interest is difficult to avoid in programs. 



'*'1988 AMS Report, supra note 91, at E-2 (quoting Joseph DePinto, Clarkson University). 

'^'^Telephonc interview with Mr. Victor Wesibroolc, supra note 123. 

'"See Chubin & Hackett, supra note 3, at 203-04. Chubin and Hackett argue: 
Reviewers and referees should sign their reviews and should be openly associated with the work they approve or disapprove. This would 
hold reviewers publicly accountable for their decisions and would take a step toward acknowledging the value of reviewers' work. No 
longer would it be convenient for a reviewer to trash another's work. Nor would it be advisable to endorse unexamined work. . . . [W]e 
understand that removing the shield of anonymity may reduce reviewers' candor and put them al risk of reprisals. Bui the current practice 
is aemi-open, with authors and proposers free to speculate about their critics' identities and, inappropriately, to reply in kind. To the 
extent that some of these reprisals are certain to be in error, our proposal will eliminate such misukes. Better still, by nuking open review 
the usual practice, all would live in glass houses, not just those with tell-ule points of view, writing styles, or typefaces. Most importantly, 
the communal character of science would be openly recognized and reinforced, the tide of coven careensm would be stemmed (or brought 
into the open), and the opportunities for communication among scientists would be improved. Id. 
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like the Science and Technology Centers Program, that make very large grants to a small number of 
applicants. One researcher observed that virtually every major research institution in the country 
either has or is applying for one of these large awards. It is therefore difficult to find qualified 
scientists for mail reviews who are not associated with a competing institution. Every outside 
reviewer from a competing institution knows that the chances of his or her institution receiving one 
of the 10-15 annual awards are higher if proposals from the other institutions are downgraded. The 
fact that the reviewer may not participate in the consideration of the reviewer's institution's proposal 
does not prevent the reviewer from conmienting adversely on remaining proposals. To some extent, 
this potential conflict of interest can be avoided by seeking out scientists from industry, and by 
careftilly monitoring the meetings of the multiple panels. 

As in NIH, it is possible for an application to be downgraded by a reviewer who feels threatened 
by the proposed research. One NSF-ftinded researcher noted: 

Sometimes your proposal is an alternative hypothesis to someone's work. If they 
don't like it or it threatens their work, they'll strike it down. Program directors 
usually oversee this, but they are too busy and sometimes this bias leads to rejected 
proposals. 

Even after implementation of the promised changes, it is not clear that NSF's conflict of interest 
policies will prevent research rivals and colleagues of research rivals from reviewing proposals. 

None of the NSF officials or researchers interviewed for this report knew of any instances in 
which an applicant claimed that his or her ideas were appropriated by a reviewer. They suggested 
that, like cases of plagiarism, instances of stolen ideas would be referred to the agency's Inspector 
General . 

13. Peremptory Challenges 

The typical project grant program in NSF allows two forms of peremptory challenges. First, 
applicants may request the long list of potential reviewers that each Program prepares and request 
that his or her proposals not be sent to specific persons on that list. If the applicant provides good 
reasons for the request, the Program Officer usually honors it. Second, after a rejected applicant 
reads the comments of the reviewers, he or she may request that a particular reviewer not be allowed 
to review his or her applications in the ftiture. Once again, reasonable requests are usually honored. 
Thus, peremptory challenges are allowed in the context of a system in which reviewer anonymity is 
preserved. 

14. Lobbying 

Like NIH, NSF encourages applicants to stay in frequent contact with the NSF staffer assigned 
to his or her proposal. Most successful applicants believe that this is very sound advice. The staff 
can offer valuable suggestions to enhance a proposal's chances of success. The primary difference 
between NSF and NIH lies in the power of the lower level staff to affect the outcome of the 
decisionmaking process. Whereas NIH staffers have very little influence over substantive outcomes, 
lower level NSF staffers are quite influential, because they make the initial rankings. On the other 
hand, the discretion of the NSF staff is considerably limited by the quantitative nature of the scoring 
by the outside peer reviewers. As in NIH, it is difficult to justify departures from the ranking 
determined by averaging the reviews. None of the NSF staffers and researchers interviewed for this 
report reported instances of rejected applicants' lobbying upper level officials at NSF to overturn 
funding decisions made by Program Officers on the basis of mail reviews. Since the identities of 
mail reviewers are kept strictly confidential, especially during the time that the actual review is being 
carried out, it is virtually impossible for an applicant to bend the ear of a peer reviewer. 
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15. Political Pressure 

As with NIH, outside overtures to NSF usually come in the form of easily answered status 
inquiries from congresspersons. Lower level NSF staffers did not report any overt attempts to 
pressure the agency to fund individual applicants. The practice is, in fact very much frowned upon 
in the research community. According to one NSF-funded researcher, "This is a touchy subject. 
Generally, applicants don't want outside pressure, even if it's for them because it's looked down 
upon." Applicants who might be tempted to bring outside pressure to bear on the process know that 
the tactic may backfire as future panels place proposals from such pushy researchers at the bottom of 
their lists. Even in the Science and Technology Research Centers Program, in which very large five- 
year grants are made to less than fifteen institutions per year, there is very little outside political 
pressure. In any event, the reviewers, who have the primary responsibility for evaluating the 
proposals, are shielded from outside pressure by upper level NSF staff. For example, the staff does 
not even allow applicants to have fancy receptions for site visit teams. 

16. Efficacy of Appeal Procedures 

Since a proposal can usually be resubmitted within months of the first rejection, researchers are 
far more likely to accept the agency's standing offer to revise and resubmit declined applications than 
they are to invoke the possibly lengthy appeal process. The NSF response to the Public Citizen 
petition noted that the appeals process was rarely used in the past; its refiisal to allow an opportunity 
for rebuttal prior to the grant awards will ensure that it will continue to be rarely invoked. 

17. Program Evaluation 

In 1989, NSF established a program for peer review of its peer review process. Each 
Direaorate NSF appoints a "Committee of Visitors" to examine the agency's files and evaluate the 
peer review process in operation at three-year intervals. The Committees must be "balanced in terms 
of its views on programmatic issues and in regards to the institutional, geographic and personal 
characteristics of its members."'^ At least two members may not be serving on any NSF advisory 
committee and may not have been applicants to the program under review for at least five years."' 
Each review must address: "(a) the integrity and efficiency of processes used to solicit, review, 
recommend and document proposal actions; (b) the relationship between award decisions and 
program goals; and (c) program support of Foundation-wide initiatives." Other topics that should be 
addressed where relevant include whether "an adequate number of high quality reviewers with 
technical competence and freedom from bias" were selected for each proposal and whether the 
program demonstrated a "balance of awards in terms of subject matter, size versus number of awards, 
and age, experience and geographic distribution of principal investigators.""" Although the reviews 
are generally positive,"' these systematic "audits" have the potential to detect instances of bias in the 
grantmaking process. The certainty that the records will be audited every three years should also 
serve as a prophylaxis against bias. 



'''^NSF Manual 1, "Administrative Information Manual (ADM),* ch. Vni. §340, updated January 31, 1991. 
"'Id. 

""Id. §350. 

"*lnlerview with Mr. James McCuUough, Program Evaluation SlafT Director, National Science Foundation, Washington, D.C., April 
3. 1992. 
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IV. PEER REVIEW IN THE ENVIRONMENTAL PROTECTION 
AGENCY'S OFFICE OF EXPLORATORY RESEARCH 

The United States Environmental Protection Agency (EPA) is one of the largest regulatory 
agencies in the United States with more than 19,000 employees located in its Washington, D.C. 
Headquarters, its ten regional offices and several research laboratories throughout the country."* 
Unlike NIH and NSF, EPA's functions are primarily regulatory in nature; it writes and enforces 
standards aimed at protecting and enhancing environmental quality. EPA does, however, have a 
research mission to learn more about the impacts of human aaivities on the environment, and it has 
particular needs for research on issues that arise during its standard -setting activities. Although it 
can never hope to conduct or fund all of the research necessary to provide an adequate scientific basis 
for its rules and regulations, it attempts to identify in advance particular scientific questions that may 
arise in future rulemaking initiatives and to conduct research on aspects of those questions to help the 
regulatory staff decide how to write supportable standards. Most of EPA's research budget is spent 
on research carried out in its own laboratories, but the agency does have a substantial budget for 
supporting outside research in universities, corporate laboratories and private consulting 
companies.'^ 

A. Discretionary Grant Programs 

Most external EPA-funded research is administered through the agency's Office of Research and 
Development (ORD), which is headed by an Assistant Administrator who is appointed by the 
President with the advice and consent of the Senate."* That Office is in turn divided into eight 
Offices, each of which is headed by a career official in the Senior Executive Service.'" Each of these 
Offices administers some investigator-initiated discretionary grants, and ORD as a whole funds 
approximately $40,000,000 per year in research grants.'" This Report will focus on the Office of 
Exploratory Research (EPA/OER), because it "is the primary contact between the Environmental 
Protection Agency and the environmental research community" and because it has traditionally relied 
heavily upon peer review in awarding research grants.'*' 

The Office of Exploratory Research (OER) was created "to support research aimed at developing 
a better basic scientific understanding of the environment and its inherent problems." Its primary 
objectives are: 

To have the environmental research community aware of and working on problems of 
interest to EPA; 



'^^The primary EPA research laboratories are located in Cincinnati, Ohio and Research Triangle Park, North Carolina. 1( has smaller 
laboratories in Las Vegas, Nevada, Ada, Oklahoma, Athens Georgia, Corvallis, Oregon, Duluth Minnesota, Gulf Breeze, Florida, and 
Narragansett, Rhode Island. See United Sutes Environmenul Prouction Agency, FY-1991 EPA Research Program Guide 127-29 (I99I) 
(hereinafter cited as 1991 Program Guide]. 

'^In 1986, for example, EPA awarded SI0.6 million for investigator initiated research out of a total extramural budget of $217 million 
for environmental and related research. 1988 AMS Report, supra note 91, at D-1 . 

'''*40 C.F.R. §1.45. See Government Manual at 556. 

'^^The eight offices in the Office of Research and Development are: the Office of Technology Transfer and Regulatory Support; the 
Office of Exploratory Research; the Office of Environmental Engineering and Technology I>emonstration; the Office of Health Research; 
the Office of Environmental Processes and Effects Research; the Office of Modelling, Monitoring Systems and Quality Assurance; and the 
Office of Health Environmental Assessment and the Office of Research and Program Management. See 40 C.F.R. §1 .45. 

'**CFDA at 877. 

"'Steven Smith & Stever Klein, Management Study of the Office of Exploratory Research 1 (1990) [hereinaf\er cited as OER 
Management Study]. See als< .Jnited Slates Environmental Protection Agency, Orientation Handbook for Members of Research Grants 
Science Review Panels 3 (1991) [hereinafter cited as Orientation Handbook]. OER's niche is defmed by research objectives and priorities 
different from those of other federal progranu supporting investigator-initiated environmental research. The grant program is interesud in 
research which holds some promise for improving EPA's ability to delect and reduce the impact of environmental pollutants on human 
health and the environment. The scope of this interest encompasses the transport, transformation, faie, effects, and control of these 
polluunU. 1988 AMS, supra note 91, Report at 15. 
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To have the environmental research community aware of and working on problems of 
interest to EPA; 

To promote close interaction and mutual awareness between EPA researchers and the 
environmental research community; and 

To provide general support to the research community for work on fundamental 
environmental research, thereby promoting a solid foundation of knowledge and a 
cadre of scientific and technical personnel in the environmental sciences.'*^ 

Most OER-supported research is fundamental, long-range research aimed at emerging 
environmental problennis or cutting edge research exploring advanced concepts and techniques in the 
enviromnental sciences and engineering. '"^ OER administers several "core programs," including a 
Competitive Research Grants (CRG) Program, an Environmental Research Centers Program, a 
Hazardous Substance Research Centers Program, a Visiting Scientists Program and a Small Business 
Innovation Research (SBIR) Program.'** This Repon will concentrate on the CRG Program and the 
SBIR Program. 

The major functions of the CRG Program are "to manage a program of long-range research 
through grants to qualified investigators and to operate a system of peer review for competitively 
selecting and awarding projects."'" The CRG Program funds research in a broad range of areas 
including identification and characterization of hazardous substances, intermedia transport of 
pollutants, pollutant fate in all media, human and ecological risk assessment, incineration, waste 
reduction, and wastewater treatment."* Examples of CRG Prog ram -funded research include studies 
aimed at exploring the relationship between solar ultra-violet radiation and skin melanomas, 
determining the effects of environmental stressors, and studying the pathology of trees exposed to air 
pollution."^ 

There are two vehicles for funding investigator-initiated research in the CRG Program: (1) 
researchers can respond to a general annual solicitation for research proposals in six broad areas of 
environmental research; and (2) researchers can respond to special solicitations for highly defined 
research in particular areas called "Requests for Applications" (RFAs).'** The RFA is a vehicle for 
soliciting "a one-time competition in a well defined high priority research area.""* In FY 1989, CGR 
Program funded 62 new grants and administered 152 active grants for a total of about $18,000,000. 
An average grant award was for 2 1/2 years at a level of about $100,000 per year.'" The program 
has sufficient funds for only about one-sixth of the applications that it receives.'" 

The SBIR Program supports innovative research with potential practical applications. EPA's 
SBIR awards are made through contracts, rather than grants, but the agency uses the same procedural 
approach to peer review in both programs. EPA devotes about $3 million per year to about 25 Phase 



"^United States Environmenul Prouclion Agency, FY 1989 Annual Report of the Research Grants Program 1 (1990) fhereinafler 
cited as 1989 Anniul Report]. 

'^OER Management Study, supra note 181, et 2. 

'^1991 Program Guide, supra note 176, at 112-13. OER also administers a manpower training program and a workforce 
development program that are not directly related to funded research. See OER Management Study, supra note 181 , at I . 

'*^United Stales Environmenul Prouclion Agency, Research Grants Program Annual Report FY 1988 (1989). 

'^1989 Annual Report, supra note 182, at 1; AMS, supra note 91, Repon at 15 ('Some academic scientists interviewed for this 
evaluation think of OER's program as the midway poini on a continuum between basic science and applied science— between NSF and NIH 
on one hand and the EPA labs on the other.*) 

'"OER Management Study, supra note 181, at 2. 

'^1989 Axmual Report, supra note 182, at 1; OER Management Study, supra note 181 . at 2. 

"'1989 Annual Repon, supra note 182. at I, 

"°1989 Annual Repon, supra note 182, at 1-5. 

'"OER Management Study, supra note 181, at 2-3. The AMS Report suggests thai the funding rale is closer to one-third. AMS 
Repon, supra note 177, al 22. If this is true, the competition is not unusually stiff. 
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Phase II funding.'" Phase I awards are limited to the statutory maximum of $50,000; phase II awards 
are limited by EPA to $150,000."* Competition is very stiff; in FY 1990, only 32 out of 435 Phase I 
applications were funded. The program currently focuses primarily upon pollution control 
technology and instrumentation research"* Examples of SBIR-funded research include a project on 
innovative incineration technology for the fiberglass industry; a study of microbial removal of 
organic sulfur from coal; and an examination of novel sensors for metal -ion detection and 
quantification."* 

B. Criteria for Awarding Grants 

The broad criteria for evaluating research proposals are as follows: 

A proposal is judged for (a) scientific merit in terms of: (1) strengths and weaknesses 
of the projea; (2) adequacy of overall project design; (3) competency of proposed 
staff; (4) suitability of applicant's available resources; (5) appropriateness of the 
proposed project period and budget; and (6) probability that the project will 
accomplish stated objectives; and for (b) program interest in terms of: (1) the need 
for the proposed research; and (2) relationship to objective(s) in an approved work 
plan.'" 

These criteria do not vary greatly from those used to evaluate the scientific merit of proposals in 
NIH. 

C. Procedures for Awarding Grants 

The grants selection process at EPA/OER consists of a "dual system of review" that relies 
heavily upon six "Science Review Panels" established on an ad hoc basis in each of the program's six 
principal areas of interest.'" EPA/OER enters into a cooperative agreement with each Panel 
Chairperson, an outside scientist or engineer appointed to a single three-year term by the Assistant 
Administrator for Research and Development, to supervise the peer review process. The chairperson 
is responsible for selecting the members of the Panel, overseeing the Panel's scientific review of the 
applications assigned to it, and providing EPA/OER staff with a Summary Statement of the Panel's 
recommendations for each application. With the approval of the EPA/OER staff, the chairperson 
chooses 20-60 members for the Panel from among recognized experts in the relevant field. Panelists 
are compensated at the rate of $200 per day or fraction of a day plus transportation and per diem 
expenses. An EPA/OER staffer called a "Science Review Administrator" (SRA), is an ex-officio 
member of the panel. The SRA's functions are to provide policy and procedural guidance and 
managerial assistance.'" 

All grant applications are initially referred to the agency's Grants Administration Division where 
they are reviewed for legal and administrative acceptability. They are then given to the Research 
Grants Program for assignment to the appropriate peer review panel. Once assigned to a panel, the 
Chairperson reviews them with the assistance of the SRA and assigns each one to at least three Panel 



"^United States Environmental Protection Agency, Small Business Innovation Research Program 2 (1990) [hereinafter cited as SBIR 
Program) . 

'»*Id.at2. 

"^Id, at 6-7; OER Management Study, supra note 181. at 6. 

'^BIR Program, Abstracts of Phase I and Phase D Awards (1989). 

'^CFDA at 878. The agency's guide for peer reviewers provides that reviewers should *focu8 attention on the scientiHc menu and 
utility or potential utility of research proposals, the competence of the principal investigator and the adequacy of institutional facilities.* 
Orientation Handbook, supra note 181, at 2. 

'^^The six principle areas of interest are: chemistry and physics of water, chemistry and physics of air, engineering, biology, health, 
and socio/economic studies. 

'"orientation Handbook, supra note 181, at 1. The Chairperson is supposed to make every effort to provide for representative 
geographic distribution and minority group representation on the panels. 
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All grant applications are initially referred to the agency's Grants Administration Division where 
they are reviewed for legal and administrative acceptability. They are then given to the Research 
Grants Program for assignment to the appropriate peer review panel. Once assigned to a panel, the 
Chairperson reviews them with the assistance of the SRA and assigns each one to at least three Panel 
members who serve as "primary reviewers" of that application. The Chairperson attempts to ensure 
that no panel member is the primary reviewer for more than five applications per cycle. If a reviewer 
believes that an application has been inappropriately assigned to him or her, it may be returned to the 
Chairperson and reassigned. Each primary reviewer evaluates the applications in accordance with the 
listed criteria and prepares a written evaluation with an accompanying recommendation to approve, 
disapprove or defer the proposal.^ All panelists are sent the cover sheets and abstracts of all 
applications that will be considered at the upcoming meeting, and any member may request a copy of 
the full proposal for any application in advance of the meeting. 

Depending upon the workload and the availability of funds. Panel meetings are scheduled two or 
three times per year. The Panel's consideration of each application begins with a review of the 
written evaluations and recommendations prepared by the three primary reviewers. A full Panel 
discussion follows until the proposal's strengths and weaknesses have been fully aired. After the 
discussion on an application is completed, each panelist "independently and privately" records a 
numerical priority score ranging from to 100 that reflects the level of scientific merit of the 
application.*' The staff then ranks the proposals according to score. The Panel may not recommend 
for funding any proposal with an average score of less than 60, and all proposals with an average 
score of greater than 60 receive an automatic recommendation. The Chairman, with the assistance of 
the SRA, prepares a "Summary Statement" setting out the Panel's recommendation and the reasons 
therefore. EPA/OER staff give the "highest consideration" to the top two to four applications for 
each Panel. The remainder are "selected in part, on criteria other than technical merit and utility, 
such as program balance and budget."^ 

Because of the very high volume of Phase I SBIR proposals, EPA/OER has developed a 
procedure for pre-screening applications to cull out clearly substandard proposals and to reduce the 
volume of proposals to a manageable level. After sorting out the incoming proposals, the SBIR 
Program Manager calls in other EPA/OER staffers and 10 outside reviewers from the relevant Panel 
for two days of culling. The SBIR Program Manager reads all of the proposals, and each outside 
reviewer reads a large group of proposals assigned to him or her. Each outside reviewer assigns 
scores and writes up brief explanations for all of the proposals assigned to that reviewer, but the 
reviewer does not read or discuss the proposals assigned to the other reviewers. Thus, instead of 
considering all of the incoming applications, each of the 10 outside reviewers only considers about 10 
percent. The SBIR Program Manager writes Summary Statements for all of the incoming proposals. 
On the basis of the scores, the SRA selects 50-70 proposals for the full three-person and full 
committee review described above. 

Grant applications, review materials, and proceedings of Panel meetings are considered 
confidential. EPA/OER 's procedures strictly forbid any communications concerning an application 
between members of the panel and applicants. All communications and inquiries must be channeled 
through the SRA or the SBIR Program Manager. In addition, the agency's Orientation Handbook for 
panelists speaks particularly to the issue of trade secrecy, providing that: 

Panel members are also bound by the confidentiality of all proprietary parts of the 
proposals they read under this guidance; no Panelists will disclose or use to their own 
advantage any data, concept, research protocol, or any other idea included in the 
applications.*' 



*^d.at2-3. 

»'ld.at3. 

^d. at 3-4. Sec also 1988 AMS Report, supra note 91 

^■'Orienlalion Handbook, supra note 181 at 5. 
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Although the Orientation Handbook is curiously silent about the confidentiality of the reviewers' 
identities, EPA/OER's policy is to never divulge the names of primary reviewers to anyone, 
including rejected applicants. No attempt is made, however, to keep ^e attendance list for particular 
panel meetings secret.^ Thus, while a rejected applicant cannot ascertain the identities of the three 
persons who reviewed his or her proposal, a persistent applicant can fmd out who probably sat on the 
Panel that evaluated the proposal. 

The Orientation Handbook also speaks to the agency's Privacy Act obligations.^ Immediately 
following a Panel meeting, all applicants who "failed" the review receive a rejection letter. Those 
whose applications "passed" the technical review receive a letter informing them of that fact and 
telling them that their applications are under consideration by the EPA/OER staff. After the final 
funding decisions have been made, those candidates not selected receive a second letter informing 
them of that fact. Five or six weeks after this, every applicant receives the Summary Statement that 
consists of the Chairman's summary of the Panel's technical review of the application.^ The 
Summary Statement is regarded as confidential and is provided only to the applicant. Applicants are 
later allowed access to all documents generated during the review of their grant applications, 
including the reviewers' written comments, if available. "Reviewers* written comments and scores, 
however, are not retained after their substance has been incorporated into Summary Statements."^ 

EPA/OER's conflict of interest requirements provide that no panelist may attend a panel meeting 
in which his or her own grant application or the application of a close relative, friend or close 
professional associate is evaluated. A panelist may attend a meeting at which an application from his 
or her own institution is considered, but must leave the room during the consideration of that 
application.™ 

Although EPA/OER has no formal procedures for challenging the agency's funding decision, it 
has established a formal appeals process under which rejected applicants may appeal the failure to 
correct any factual errors manifested in the Summary Statements or other documents in the 
applicant's file.^ According to EPA/OER staffers, disgruntled applicants are rare. When a rejected 
applicant asks the relevant SRA why his or her proposal was rejected or attempts to challenge the 
agency's decision, the SRA typically suggests that the proposal be resubmitted for consideration 
during the next funding cycle. The resubmittal will be considered by the same panel, which normally 
contains about 50-60 percent of the people who reviewed the original panel. 

D. The Process in Operation 

EPA/OER's research grant process relies heavily upon peer review. Outsiders choose the peer 
reviewers, manage the peer review meetings, and evaluate and rank the proposals. Although the 
ultimate funding decisions are made by EPA/OER staffers, they rarely depart from the Panel 
recommendations. According to one outside management review of EPA/OER's funding process, 
"[a] combination of tailored review groups, longer-than-average discussion time, and stringent 
criteria produces a very rigorous screening of proposals on the basis of scientific merit. "^'° As with 
the other programs examined in this report, however, the process in EPA/OER does not always 
operate in practice exactly as described on paper. 



^'^oinmenu of Mr. Robert A. Papetii, Director, Research Grams StafT, Offtce of Research and Development, EPA on an earlier draft 
of this Report, August 28, 1991. 

^^EPA also has promulgated ptTKedural regulations implementing the Privacy Act. 40 C.F.R. pt. 16. 
''"Commenu of Mr. Robert A. Papeni, supra note 204. 

Orienution Handbook, supra note 181, at 5. 
^Id.at 1. 
^40 C.F.R. §16.9. 
^'°1988 AMS Report, supra note 91, at 22. 
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1. Feedback and Rebuttal 

EPA's Privacy Act regulations appear on the surface to provide quite a bit of feedback to 
rejected applicants. In theory, applicants may see all documents generated during the review of their 
grant applications, including site visit reports, Summary Statements, and reviewers' written 
comments, but only if they are still available when the request is made. Since the all written 
documentation is discarded after the Summary Statement is drafted, as a practical matter applicants 
cannot secure verbatim copies of reviewers' written comments, and they never see their scores. 2" 

2. Favoritism 

EPA/OER attempts to avoid favoritism, animus and conflicts of interest by setting aside a period 
of time at the first of each Panel session to a discussion among the panelists of these issues with 
respect to the pending applications. Since the panelists tend to be well-established university research 
scientists with substantial teaching careers, the discussion often focuses on proposals from their 
former students. The agency has adopted a rough rule of thumb that a panelist need not recuse 
himself from the consideration of proposals from former undergraduate students or from former 
graduate students who have not graduated within the last five years. 

3. Old Boy Network 

EPA/OER officials acknowledge that the history of discretionary research grants in EPA is not 
inspiring. Prior to the establishment of the Office of Exploratory Research in 1980, research grants 
were awarded by EPA staff in an ad hoc fashion. ^'^ The system was a prototypical good old boy 
network in which agency staffers awarded grants to their favorite principal investigators with little 
regard to the peer review that was conducted and little threat of subsequent evaluation. As a 
consequence, the agency's reputation as a research institution suffered, and a task force within ORD 
was created to establish a more effective peer review system for awarding grants. 

The peer review system was barely underway, however, when the recently appointed Assistant 
Administrator for Research and Development began to abuse it. In one of the lower profile 
contretemps of the Gorsuch Administration at EPA, Assistant Administrator Andrew Jovanovich was 
accused of steering grants to one of his former business associates outside of the peer review process. 
According to a report by the agency's Inspector General, a Massachusetts Institute of Technology 
researcher, Ronald Probstein, met personally with Jovanovich after his grant proposal to study water 
treatment technology was rejected by one of the recently formed peer review panels. Jovanovich had 
previously supervised Probstein 's work at a private research institute. After Probstein angrily 
denounced the Panel's decision and complained that he would have to lay off several graduate 
students, Jovanovich interceded with the lower level staff and demanded that Probstein's proposal be 
reviewed by another peer review panel containing only two members, one of which would be selected 
by Jovanovich and the other by Probstein. Not surprisingly, the new panel was effusive in its praise 
of the proposal, and a $77,000 grant was awarded. Jovanovich explained that lower level EPA 
grants officials were "very poor at selecting the right work," but the Inspector General found that 
"allowing a scientist to select his own peer reviewer, especially one he has previously worked with, 
lacks any appearance of independence or objectivity . . . and would leave EPA open to charges of 
cronyism. "^'^ Later, Jovanovich 's appointment as Assistant Administrator for Research and 
Development was withdrawn, and he was reassigned to a low-level agency post. 

In the intervening years since the Jovanovich incident, there have been no further charges of 
impropriety at high levels in the Office of Research and Development, and the extensive peer review 
process for awarding grants in OER has apparently proceeded without substantial interruptions from 
high levels. Although the program is no longer criticized for cronyism, it is still possible for the 



^"Commcnls of Dr. Robert A. Papetii, supra note 204. 

"^'^ee M. Landy. M. Roberts & S. Thomas, The Environmenul Protection Agency 50 (1990). 

^'^Howie Kurtz, EPA Research Chief Violated Contract Award Rules. Probe Finds. Washington Post. April 10, 1982, A3. 
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Panel Chairperson to select panel members from among a fairly narrow cut of researchers who are 
familiar with one anothers' work and who have similar views about how research ought to be done. 
The Director of the Research Grants Staff, however, reported that he had no reason to believe that 
the Chairpersons behave this way, and he pointed out that one of the functions of the SRA is to 
ensure that they do not function as old boy networks. SRAs are told to insist that panel membership 
reflect a diversity of specializations, talents, and points of view. Since the SRA must ultimately 
concur in the Chairperson's selection of Panel members, they have the power to police the system 
against "old boyism."^'* 

A further hedge against old boyism is inherent in the very nature of the program. Because of 
budgetary and personnel limitations, EPA/OER can only assemble six panels to review proposals 
coming from all areas of environmental research. ^^ Since EPA must of necessity recruit experts from 
widely divergent fields, there is less risk of any particular panel becoming dominated by a group of 
panelists hewing to a narrow view of proper science and legitimate scientific subjects. 

4. Halo Effect 

EPA/OER staffers are ambivalent about the "halo effect." One SRA said that he did not believe 
that the halo effect seriously affected the process at EPA/OER, noting that: "We've shot down many 
big names." Another SRA noted that many successfully funded applicants are funded very 
frequently, and suggested that: "There are some applicants with more clout than others." He opined 
that "a former [Panel] chairman may pull some weight." Yet the fact that during the years 1986-90 a 
total of 281 awards were made to a total of 135 institutions, no one of which received more than 4.5 
percent of the total funds awarded suggests that the money is spread around fairly evenly. 2'* 

Established researchers have a much better chance of obtaining funding from EPA/OER than 
"youngsters" who are new to the system. One EPA/OER staffer noted that because of the intense 
competition for grants, it is often very difficult for a new researcher to obtain funding. Given a very 
limited pool of resources, the senior scientists on the Panels no doubt tend to recommend funding for 
proposals from other well-established researchers, because they are more likely to yield useful results 
than those of novices. No special effort is made to direc* resources to younger researchers just 
entering the field. The overall result may be that discouraged newcomers opt out of the system and 
pursue research in other areas or with other funding agencies. 

5. Personal Animus 

At the first of every Panel meeting, panelists are provided an opportunity to identify proposals 
from researchers about whom the panelists have strongly negative views. On one occasion early in 
the history of the program, a panelist expressed a strong personal dislike of an applicant, and he was 
happy to be relieved of the obligation of participating in the consideration of that applicant's 
proposal. This approach, of course, relies entirely upon the candor of the panelists in identifying any 
animus that they might harbor toward particular applicants. Perhaps not surprisingly, this happens 
only very rarely.^" 

6. Mavericks 

As in NIH and NSF, peer review committees in EPA/OER tend to look unfavorably upon 
maverick proposals that reject the conventional wisdom of the research community. According to 
one EPA/OER staffer, "Panels are basically very conservative and want to have some assurance of 
success. They are not inclined to fund way-out proposals." The staff, however, makes a special 



^'*CommcnU of Mr. Re - A. PapeUi, supra nolc 204. 

^'*ld. By comparison, EPAyOER convenes onJy one panel on hunruin health, while NIH has over 100 study sections devoted to that 
topic. Id. 
2'*ld. 
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effort to ensure that unusual, but innovative proposals are fairly considered. The difficulty, of 
course, lies in distinguishing an innovative but promising proposal from a quirky idea that has no 
chance for success. According to the Director of the Research Grants Staff: 

It is our impression that many of the proposals received (particularly in the SBIR 
program) are not the productions of inventive genius not recognized in its time, but 
inoperable schemes, reinventions of old ideas, and theories which violate basic 
principles of physics, chemistry, etc. The authors of these proposals may well be 
viewed as mavericks fighting against the weight of conventional scientific opinion, 
but they are precisely the reason why we use technical and scientific peer panels to 
review proposals and recommend for or against support by EPA.^'* 

7. Tunnel Vision 

Since the Panel Chair is responsible for selecting the panel members, it is entirely possible that 
the committee will be composed of a homogeneous group of main-line scientists who share the 
Chairperson's views about the relevant scientific questions. This can lead to tunnel vision. 
EPA/OER formally attempts to reduce this possibility by insisting that Panel Chairs step down at the 
end of the three year terms, but in many cases the terms have been extended informally to four or 
more years. 

8. Applicant Anonymity 

EPA/OER does not attempt to keep the names and institutions of the applicants from the panel 
members. 

9. Reviewer Anonymity 

Although EPA/OER is very careful about maintaining the confidentiality of the Panel 
proceedings, nothing in EPA/OER 's Orientation Handbook for panelists suggests that the names of 
the panelists should not be revealed to applicants. Although the agency has a strict policy against 
revealing the names of the three primary reviewers to applicants, the attendance list for particular 
panel meetings is publicly available. Hence, a persistent applicant can ascertain the identities of the 
members of the panel that probably considered his or her application, but cannot ascertain the 
identities of the three panelists who conducted the intensive reviews of his or her proposal. 

10. Fmancial Conflict of Interest 

The EPA/OER guidelines parallel the NIH guidelines on financial conflict of interest. No 
panelist may participate in a panel meeting that evaluates his or her own grant application or the 
application of a close relative, friend or close professional associate, and Panel members must absent 
themselves from meetings when applications from their own institutions are considered. There is one 
significant difference, however, that is inherent in the limited number of panels available for 
reviewing applications at EPA/OER. In NIH, the proposal of a study section member or close 
relative or associate will go to an entirely separate panel, and if an appropriate panel with sufficient 
expertise does not exist, an ad hoc panel will be created. In EPA/OER, a panelist's application goes 
to his or her panel, but the applicant is not invited to the meeting in which that proposal is 
considered. 

Although this procedure shields the decisionmaking process from overt participation by the 
panelist/applicant, the remaining panelists know at the time that they evaluate a fellow panelist's 
proposal that they will have to work with that panelist in the future. This may inspire them to view 
the proposal favorably. Even though the scores are always kept confidential, a Panel member whose 
proposal is rejected knows that at least some of the other panelists panned it. Moreover, to the extent 
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down, this arrangement does nothing to dispel the suspicion that the granting process is an old boy 
network in which friends in powerful positions help each other out.^" 

11. Research Conflict of Interest 

Like the NIH guidelines, the EPA/OER guidelines do not prevent reviewers from judging the 
applications of direct competitors. EPA/OER 's Orientation Handbook for panelists does, however, 
speak obliquely to one aspect of research conflict of interest. Addressing the issue of confidentiality, 
the Handbook provides that "no Panelists will disclose or use to their own advantage any data, 
concept, research protocol, or any other idea included in the applications. "2=° This provision may be 
broadly construed to prevent a researcher from using information obtained during the review process 
to his or her own advantage in his or her own research, whether or not there is a potential for 
financial gain. In both instances, however, the agency has declined to adopt the prophylactic 
approach that is normally used to address conflicts of interest—viz. recusal from the decisionmaking 
body in which the conflict of interest might arise. ^2" 

12. Peremptory Challenges 

EPA/OER makes no provision for challenges from applicants to the participation of certain 
individuals in the agency's evaluation of their proposals. Since each funding program has only a 
single very large peer review committee, a true peremptory challenge in which the challenged 
reviewer plays no role whatsoever in the evaluation of the challenger's proposal or any competing 
proposals would probably be very difficult to implement. The agency, however, could relatively 
easily implement a modified peremptory challenge under which the challenged individual would not 
be allowed to participate in the discussion or scoring of the challenger's application. The potential 
for animus would, in other words, be treated exactly the same as the potential for favoritism that 
arises when the Panel considers an application from the reviewer's own institution.^ 



^'^See Part 5. In a response to an earlier draf\ of this Report, the Director of EPA's Research Granu SufT noted that EPA's panels 
differ from those of NIH in that they are itot standing conunittees with fixed memberships. Because EPA's panels are assembled ad hoc 
from an informal list of technically proficient people who have agreed to serve as panelists. In this more fluid context, subtle quids pro 
quo are not as easily arranged. In addition, because EPA strict division of disciplines among its panels arguably precludes sending a 
panelist's proposal to another panel. Unlike NIH, there are no closely related panels with the technical capacity to evaluate the proposal of 
an applicant who is a member of a different panel. Comments of Mr. Robert A. Papetti, supra nou 204. 

Oite solution to this dilenuna is simply to preclude all applicants from sitting on any panels. Since the agency has available to it a list 
of possible panelists and since a panelist need serve for only a single round of evaluations, it should be easy enough to assemble a panel 
out of persons who do not have an application currently pending before EPA. Moreover, since each panel draws on expertise in a broad 
range of disciphnes, the universe of potential panelisu is no doubt fairly large. 

^^Orientation Handbook supra note 1 8 1 , at 5 . 

^'in a response to an earlier draft of this Report, the Director of EPA's Research Granu Staff noted that: 

Whenever anybody who is proficient in a technical subject area reviews work of another person proficieni in the same or 
closely allied area then a potential conflict exists since both are competitors for ideas in the same technical field. There 
is no way out of this short of assigning an unqualified reviewer to judge the proposal. 

Comments of Mr. Robert A. Papetti. supra note 204. The validity of this argument depends upon how one defines a 'qualified* reviewer. 
If the universe of qualified reviewers consists exclusively of those persons who can understand every technical nuance of a proposal, then 
the argument may be correct. If, however, the universe includes persons with a strong grounding in science who practice in a related 
field, then the argument has less validity. 

^^In a response to an earlier draft of this Report, the Director of EPA's Research Granu Staff suggested (hat even modified 
peremptory challenges would be inappropriate in EPA's programs for three reasons: 

First of all, it appears that such a procedure could provide a way for applicants to influence the review of their own 
proposals by challenging the most competent and incisive reviewers. Second, we believe that implementing a peremptory 
challenge procedure would resuh in challenges from so n\any applicants that ii would be difficult for us to follow 
through. Finally, our system in which there are three primary reviewers among twenty to forty panel members tends to 
remove the impact of outlying votes. 
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13. Lobbying 

Despite EPA's history of direct intervention by high level EPA officials, EPA/OER staffers 
reported no instances in recent years of applicants' lobbying upper level EPA staff. The 
embarrassing experience of Dr. Jovanovich may still be fresh in the minds of upper level EPA 
officials and potential "lobbyists." 

14. Political Pressure 

EPA/OER staffers frequently receive letters from congresspersons in support of particular 
research applications. These are included in the applicants' files and are considered by the Panels. 
EPA/OER staffers never overturn panel recommendations on the basis of such letters. In fact, the 
letters slightly annoy EPA/OER staffers, because EPA/OER procedures require a response within a 
limited period of time, and this takes time out of their already busy schedules. 

15. Efficacy of Appeal Procedures 

EPA has established a formal appeals procedure for rejected applicants. Each Project Officer 
must designate a "Disputes Decision Official," from among senior EPA employees who are 
knowledgeable about EPA's assistance programs, to review and resolve disputes over grant 
applications and rejections. After consulting informally with the relevant persons, the Disputes 
Decision Official issues a written decision from which the applicant may take an appeal to the 
appropriate EPA Assistant Administrator. Any appeal must be accompanied by documentary 
evidence and briefs. Applicants may be represented by counsel in such appeals. The Assistant 
Administrator's written decision constitutes the final agency action, and the applicant may then appeal 
to a court. ^ 

Although a rejected applicant has a right to demand an informal hearing, this happens only very 
rarely. One staffer asserted that complaints occur only "once in a blue moon," and another reponed 
a "very, very low bitch rate." When complaints do occur, the SRA's typically advise the rejected 
applicants to revise and resubmit their proposals. None of the staffers interviewed for this report 
could remember a case in which a rejected applicant successfully went over the head of the SRA to 
the upper level management in Office of Research and Development or the Office of the 
Administrator. 



CominenU of Mr. Robert A. Papetti, supra note 204. The Tint objection is a troubling one, because it suggesu that a savvy applicant 
could 'game* the system by exercising peremptory challenges against the most critical reviewers, leaving them available to criticize 
competing proposals. If the pool of potential reviewers is very snull, this could constitute sufficient reason to deny peremptory challenges. 
The potential for abuse, however, should be greatly reduces by making the applicant explain his or her reasons for challenging a potential 
reviewer. 

The second and objections are of less concern. It would take very limited staff lime to follow through on peremptory challenges, even 
if it involved reading and evaluating two or three challenges from each applicant. In practice, the vast majority of applicants will probably 
decline lo exercise any peremptory challenges, and the others can be evaluated in relatively short order. Assigning the primary reviewing 
role to three of the twenty or so panelists does not solve the problem that peremptory challenges are supposed to solve, and may in fact 
exacerbate it. An applicant who is concerned that a panelist might harbor animus against him or his institution is not likely to be assuaged 
by the assurance that the likelihood is small that that panelist will be among the three pnmary reviewers. If the panelist really harbors 
animus, then he or she will attempt to become a primary panelist for that proposal, and if that nuneuver is successful, his or her animus 
could play a very large role indeed in the evaluation of the application. 

^^United Stales Environmental Protection Agency, Assistance Administration Manual 35-1 (1984). Somewhat more formal 
procedures are applicable to rejected applicants for SBIR grants, because EPA administers such grants as contracts subject to the Federal 
Acquisition Regulations. Comments of Mr. Robert A. Papetti, supra note 204. 
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V. PEER REVIEW IN THE NATIONAL ENDOWMENT FOR THE ARTS 

The National Endowment for the Arts (NEA) is part of the National Foundation on the Arts and 
the Humanities, which was created as an independent agency by the National Foundation on the Arts 
and the Humanities Act of 1965.^ Its mission is to "foster the excellence, diversity and vitality of 
the arts in the United States," and to "help broaden the availability and appreciation of such 
excellence, diversity and vitality," without "imposfing] a single aesthetic standard or attempt[ing] to 
direct artistic content."^ NEA is an essential part of a relatively brief tradition of government 
patronage of the arts dating back to the New Deal's Works Progress Administration.^ Since its 
creation in 1965, NEA's budget has mushroomed from about $2.5 million to $174 million in FY 
1991.^ By almost any measure, NEA has been enormously successful in fostering the fine arts in 
the United States. On its twentieth anniversary in 1985, NEA received an Oscar for "its dedicated 
conmiitment to fostering artistic and creative activity and excellence of human genius."™ The 
inevitable tensions inherent in its goals, however, have frequently placed the agency in the public 
spotlight where, perhaps to a greater degree than the scientific agencies, the pros and cons of peer 
review have been extensively debated. 

NEA fulfills its mission primarily by awarding discretionary grants to individual artists and 
nonprofit artistic organizations through peer reviewed competitions. Since awards to organizations 
must generally be matched from some other source, the agency can serve as "a catalyst to promote 
the continuing diversity, vitality and excellence in the arts in America and to provide access to, and 
appreciation of, such diversity, excellence and vitality."^ In fiscal year 1989, NEA funded 4,458 of 
17,879 grant applications for a total of $153 million. Individual artists received 7.6 percent of the 



22*20 U.S.C. §95l.el>eq. 

22^Nalional Endowment for the Arts, Guide to the National Endowment for the Arts 2 (1990) (hcreinafler cited as NEA Guide). 
According to the NEA Statement of Mission, NEA activities are designed to: 

Demonstrate national recognition of the importance of artistic excellence; 

Provide opportunities for artists to develop their talents; 

Assist in the creation, production, presenuiion/exhibilion of innovative and diverse work that has potential to affect the 
art form and directly or indirectly result over lime in new an of permanent value; 

Assure the preservation of our cultural heritage; 

Increase the performance, exhibition and transmission of an to all people throughout the nation; 

Deepen underelanding and appreciation of the ans among all people nationwide; 

Encourage serious and meaningful an programs as pan of basic education; 

Stimulate increasing levels of nonfederal suppon of the arts; 

Improve the institutional capacity of the best of our aru organizations to develop, produce, present, and exhibit bold and 
varied fare; and 

Provide information about the arts, their artistic and financial health and the state of their audiences. 

22*See William J. Lanquette, The Federal Govemmeni-The Patron Saint of the Arts, 10 National Journal 1288 (Aug. 12, 1978); 
22^Commenu of Michael McLaughlin, Senior Staff Assisunt, Office of the Deputy Chairman for Programs and Ana Steele, Associate 
I>eputy Chairman for Programs on an earlier draft of this repon, September 17, 1991 . See also Lanquette, supra note 226. 

22*National Endowments for the Arts is Honored as 20th Birthday Nears, New York Times, Sept. I, 1985, sec. 1, pt. 2, p. 68, col. 1 . 
22^EA Guide, supra note 225, at 2. 
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monies, and the remainder went to organizations with matching grants. Almost 90 percent of the 
grants were for less than $50,000.^ 

The National Council on the Arts, composed of the NEA Chairman and 26 other members 
appointed by the President with the advice and consent of the Senate, advises the Chairman with 
respect to policies programs and procedures, reviews applications for financial support, and makes 
recommendations on funding decisions. Its function is therefore very much like the Advisory 
Councils in NIH upon which it is apparently modeled. It members must be citizens who "(a) are 
widely recognized for their knowledge of, or expertise in, or for their profound interest in, the arts 
and (b) have established records of distinguished service, or achieved eminence, in the arts."^' As 
the agency has expanded the range of artistic areas that it is prepared to support, the Council has 
become less involved in advising the Chairman with respect to individual grants and has played a 
broader advisory role with respect to more important policy issues. ^^^ 

A. Discretionary Grant Programs 

Like NIH and NSF, NEA has major funding programs in several broad areas, including Dance, 
Design Arts, Folk Arts, Literature, Media Arts (Film/Radio/Television), Museums, Music, Theater 
and Visual Arts, and Challenge and Advancement Grants. Because the agency is only able to fund 
about 25 percent of the applications that come before it, the competition in all of the programs is 
fairly stiff. This report will focus primarily upon the programs in Music, Visual Arts and Challenge 
Grants. 

1. Music 

The Music Program awards grants to assist exceptionally talented individuals and a wide range 
of organizations including professional symphony orchestras, contemporary music ensembles, jazz 
groups, national service organizations, choruses, chamber music ensembles, music festivals, and 
others. Funds are not available, however, for capital expenditures such as the construction or 
rehabilitation of facilities, summer music camps, publication costs, and purchase of musical 
instruments or uniforms.^' The program awards about $15,500,000 to about 800 successful 
recipients from a pool of around 1800 applications. Grants for individuals range from $2,000 to 
$25,000; grants for organizations range from $3,000 to $286,000. Grants are awarded to support 
"the creation and performance of music, with an emphasis on assisting the growth of American music 
and musicians."^ Examples of such grants include funding for a summer music festival, assisting an 
orchestra to include contemporary and American works in its repertoire, supporting a fellowship to 
compose a work for string orchestra and chorus, and supporting regional touring jazz clinics. ^^ 

2. Visual Arts 

The Visual Arts Program awards grants to "individuals of exceptional talent working in a wide 
range of styles and media," including painting, sculpture, photography, crafts, printmaking, drawing, 
artists books, video, performance art, conceptual art, and new genres. Awards to organizations are 
intended to "encourage individual visual artists' development and experimentation." The Program 
also supports "dialogue between visual artists and the public."^ The Program awards about $6 
million in relatively small fellowships and grants. Individual fellowships and grants are awarded at 



^^''^National Endowment for the Arts, Facu ebout the Aru Endowment (1990). 
^' NEA Guide, supra note 225, at 3. 

^^The Independent Commisgion on the National Endowment for the Aru, A Report to Congress on the National Endowment for the 
Arts 24 (1990). 

^^CDFAat746. 

^^NEA Guide, supra note 225, at 35. 

^^CFDA at 747. 

^'^NEA Guide, supra note 225, at 53. 
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levels of either $15,000 or $20,000, and organization matching grants range from $5000 to $50,000. 
Out of about 5500 applicants, the Program makes approximately 450 awards, a success rate of only 
around 8 percent.^' The Visual Arts Program is subdivided into three major categories. About $3 
million of the budget is devoted to about 200-250 individual fellowships. Less than 5 percent of the 
5000 applications are funded. In the second category, NEA awards about $2 million per year to 
visual artists' organizations; about one-half of the 200 applicants are funded. The third category 
consists of visual artist forums and art in public places. About $700,000 is devoted to funding about 
60 grants out of around 75 applications. Around $200,000 is left over for special projects. 
Examples of funded projects include: support for commissions for artists to create a permanent 
installation at a convention center, support for production of a series of videotapes on fine art 
lithography, and support for a series of lectures on art practice and critical writing by visual arts 
professionals from diverse professional and cultural backgrounds™ Other, more notorious, examples 
of projects funded by the Visual Arts Program, such as the Mapplethorpe and Serrano exhibitions, 
will be discussed in more detail in this Part. 

3. Challenge Grants 

The Challenge Program was created to advance the arts over the long-term and thereby to 
complement the work of the other programs by "offering major one-time grants for activities that 
look beyond current needs and programming." The overall goal of stimulating long-term public and 
private investment in the arts is accomplished by requiring that grants be matched on at least a three- 
to-one basis. Challenge III grants are intended to assist institutions (including arts institutions. State 
and local arts agencies, regional organizations and others) to undertake new or substantially 
augmented projects that meet one or more of the following objectives: 

1) Artistry: to assist nationally significant artistic achievement of the highest 
quality in one or more of the art forms; 

2) Access: to assist improved and broader access to the arts of quality; 

3) Appreciation: to assist deeper and broader education in and appreciation of 
the arts of the highest quality; 

4) Nonfederal support: to assist nonfederal public and private support systems 
for the arts to address any, or a combination of, the above objectives. 

During its earlier history (Challenge I and Challenge II), the Program attempted to increase the 
financial stability of existing arts institutions. Challenge III has the more ambitious objective of 
supporting "projects designed to have a lasting impact that can help move the Nation forward in 
achieving excellence in the arts, access to, and/or appreciation of such excellence."^' 

Like the SBIR Programs in the scientific agencies, Challenge III awards have two phases. 
During the first phase, NEA supports up to 50 arts organizations for an intensive one-year planning 
process during which NEA provides financial and technical assistance in drafting a long-range plan. 
Organizations that complete Phase I may apply for large grants to help implement the strategies 
identified in the long-range plans. With a budget of about $18 million, the Program supports about 
25 out of 120 applications with grants that range from $50,000 to $1,(X)0,000 apiece and average 
around $400,000.2* One example of a frinded project is a $300,000 Challenge III Grant to the New 
England Foundation for the Arts to increase support for and provide access to outstanding talent in 



23'CFDAal 750-51. 

™CFDAat751. 

^'^NEA Guide, supra nolc 225, at 63; CFDA at 754. 

2*CFDA at 755. 
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contemporary and folk arts through the creation of two new presenter networks: the Contemporary 
Arts Network and the Traditional Arts Network.^' 

B. Criteria for Awarding Grants 

The agency's statute provides some broad criteria for awarding grants. Grants are to be made to 
individuals "of exceptional talent engaged in or concerned with the arts" for the purpose of 
supporting projects and productions with "substantial national or international artistic and cultural 
significance," and that meet "professional standards or standards of authenticity or tradition, 
irrespective of origin, which are of significant merit and which, without such assistance, would 
otherwise be unavailable . . . ."^ In awarding grants, the agency is to give emphasis to "American 
creativity and cultural diversity and to the maintenance and encouragement of professional 
excellence."^ 

Grants in the Music and Visual Arts Programs are made to individuals and nonprofit 
organizations on the basis of three broad criteria: artistic quality, the project's merit, and the 
applicant's capacity to accomplish the project. The criteria for awarding Challenge III grants include 
the above criteria plus "fundraising capability, impact, readiness, and ability to sustain 
performance."^ Each funding category within each Program has its own detailed "review criteria" 
that are articulated in annual application guidelines. Although there is a good deal of variance across 
the categories within a Program, there are also many similarities. For example, the criteria in the 
Music Program for the "Choruses" category are first, and most importantly, "[t]he quality of the 
chorus' performance as demonstrated by the sample cassette" that must be submitted with the 
application. For those applicants that pass this review, the reviewers consider: 

• Quality of repertory and commitment to American works. 

• Merit of the project and the applicant's ability to accomplish the project. 

• Quality of management (including fiscal responsibility and evidence of earned 
and contributed income where appropriate). 

• Educational outreach activities and efforts to involve minorities and special 
constituencies in regard to audiences, artists, and repertory. ^^ 

The criteria for "Visual Artists Organizations" in the Visual Arts Program are: 

• Quality of artists' work as demonstrated by visual documentation of recent 
and proposed programs, 

• Quality, innovation, and timeliness of current and proposed activities. 

• Evidence that visual artists have an integral role in policy development and 
programming. 



^'CFDAat755. 

"^45 U.S.C. §954(c)(l), (2). 

^^45 U.S.C. §954(c)(l). 

In making grants, the chaimun, with the advice of the grant advisory panels and the National Council was to consider, among other 
factors, artistic and cultural significance, sundards of professional excellence, and the development of appreciation and enjoyment of the 
arts by citizens. Congress has also, over time, added other considerations, such as reaching and reflecting the culture of minority, inner 
city, rural or tribal conununities. 

The Independent Commission on the National Endowment for the Arts, supra note 232. 

^CFDAat755. 

^**National Endowment for the Arts, Music Ensembles Application Guidelines FY 1991 19 (1990). 
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• Commitment to payment of professional fees to visual artists. 

• Evidence of due consideration of women and individuals of racially and 
culturally diverse communities. 

• Appropriateness and adequacy of facilities and equipment. 

• Constituency and community served. 

• Evidence of applicant's ability to administer the project, including 
organizational stability and appropriateness of budget for the proposed activities. 

• Evidence of sound management, planning, and development policies.^ 

Consistent with the greater resources at stake, the review criteria for Challenge III grants are 
even more detailed and fall under four general headings, each of which contains several subheadings. 
The general criteria and some of the subheadings are: 

Criterion # 1 Quality of the project 

A. Artistry: The project's potential for achieving the highest artistic quality of national 
significance and lasting impact on the art form or excellence in it. 

B. Access: The project's potential to develop improved and broadened access, on a long-term 
basis, to the arts of quality. 

C. Appreciation: The project's potential, on a long-term basis for developing deeper and 
broader education in and appreciation of the arts of highest quality and of cultural significance. 

D. Support Systems for the Arts: The project's potential, on a long-term basis, to assist non- 
Federal support systems (e.g., subgranting, technical assistance) to address any one or a combination 
of objectives A through C. 

Criterion # 2 Ability to complete the project successfully 

Criterion # 3 Ability to meet the match [reference to matching grants requirement] 

Criterion # 4 Where appropriate, indications that the project can be sustained in the future 
without Federal support 

The foregoing criteria suggest that management ability plays as large a role in Challenge Grants 
as artistic ability.^' 

C. Procedures for Awarding Grants 

The discretionary grant award process at NEA is very similar to that of NIH, upon which it is 
modeled.^ As in NIH, an NEA grant application goes through a "dual review" in one of more than 
90 review panels and in the National Council on the Arts. Prior to submitting an application, a 
prospective applicant may telephone or meet with an agency staffer (called a Program Specialist) to 
discuss the proposed project and obtain feedback on how best to formulate the proposal.^ When an 
application arrives at NEA, it is "logged in," a process in which basic information from the 



^^ationat Endowment for the Am, Application Guidelines FY 1991 8 (1990). 

^^One high level official in the Challenge Grant Program suggested: *If an organization is terrific at art but not well nunaged, then it 
won't be funded.* 

^**The following discussion of the procedures in place in NEA draws on two primary sources. National Endowment for the Arts, 
Summary of Steps in the Current Application/Grant Process (manuscript April 3. 1990) [hereinafter cited as NEA Summary]; National 
Endowment for the Arts, Panel Study Report (1987) [hereinafter cited as [Panel Study Report]. 

^'According to one NEA official; 'We do a good bit of handholding - the applicants can come in and gel help.' 
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application is entered into a computer and an application number assigned. Applications are then 
forwarded to the appropriate Program Office,^ For applications for which site visits are required, 
the Program Office selects outside experts to conduct the site visit (e.g., attending live performances, 
observing facilities, visiting with applicants, etc.) and report back to the staff."' Most applications, 
however, can be evaluated on the basis of other "artistic evidence," such as manuscripts, slides, and 
tapes. 

When all of the evidence is available, it is forwarded to the relevant Advisory Panel or Panel 
section. 232 Every major NEA Program has a Panel, and some are so large that their Panels are 
broken down into Panel sections."' Each Panel or Panel section is composed of from five to fifteen 
members appointed by the Chairman of NEA (usually with heavy reliance on the Program Directors) 
on the basis of "expertise, aesthetic diversity, geographic dispersion, ethnic and gender 
representativeness, ability to serve, etc.""^ Panels are assembled from long lists of nominees from 
the council, staff, current panel members, national artist associations, the general public, and the 
White House. "^ The agency attempts to assemble panels that "include wide diversity on many levels 
appropriate to each program: different artistic and programmatic viewpoints; expertise in different 
aspects of the art form/field (different types of creative and performing artists, arts administrators, 
trustees, critics, educators, large and small organizations, traditional and experimental work, etc.); 
experience with the field in different parts of the country, different ethnic and cultural backgrounds, 
and a reasonable balance of men and women.""* Virtually all panels include representatives from a 
State or Local Arts Agency or regional arts organization. As mandated by 1990 amendments to the 
agency's statute, panels always include a member of the lay public with no particular expertise in any 
of the artistic areas covered by the panel."' 

The peer review panel system that the NEA developed very early in its history "marked the 
triumph of professional judgment over political patronage.""* All potential panel members "are 
carefully evaluated on their professional standing as well as their ability to articulate issues 
confronting their field and their willingness to make the necessary commitment of time and energy to 
prepare for and to attend panel meetings. ""* In addition, "[w]hile no panel section is large enough to 
accommodate representatives of every conceivable aesthetic viewpoint, care is taken to ensure 



^^*rogram Offices are often subdivided into several 'areas.* Each area usually has a Program Director, an Assistant Program 
Director, and one or more Program Specialists. Each Program Specialist is responsible for several subcategories of applications. The 
stafTs role is supposed to be purely ministerial; suffers are not to become involved in substantive decisionmaking. 

"'site visiu are always required for Challenge IH grants, and NEA has in fact contracted with a consultant to visit applicant 
institutions, examine their management capabilities, and report back to NEA. The consulums prepare a 3-4 page memorandum analyzing 
the applicant's fiindraising plans, the qualifications of its board of directors and staff, its historical strength and commitment to the future, 
its debt management, other aspects of its managerial abilities. 

In addition to the 'grant advisory panels' that the agency assembles to review and make recommendations of individual 
applications, it also empanels 'policy advisory panels* to provide advice on 'priorities, practices, guidelines and the allocation of resources 
for individual programs.' The Independent Commission on the National Endowment for the Arts, supra note 232, at 25-26. Members of 
policy panels have usually already served on grant panels. Id. 

National Endowment for the Arts, Arts Endowment Panels (Mimeo, March 1990) (hereinafter cited as Arts Endowment Panels). 
NEA Summary, supra note 248, at 2,; The Independent Comnussion on the National Endowment for the Arts, supra nou 232, at 
28. 

"^Comment, "Turmoil at the National Endowment For the Arts: Can Federally Funded An Survive the 'Mapplethorpe 
Controversy'?", 39 Buffalo L. Rev. 231 254 (1991). 

"*Ans Endowment Panels, supra note 253, at 2. The concern for ethnic and cultural backgrounds and balance of men and women 
came after 'the system of deference no establish professionals came under fire for perpetrating elite domination of the cultural spheres.* 
Comment, Standards for Federal Funding of the Arts; Free Expression and Political Control, 103 Harv L. Rev 1969 (1990). In 1985, 
Congress directed that NEA select panelists 'who broadly represent diverse characteristics in terms of aesthetic or humanistic perspective, 
and geographical factors, and who broadly represent cultural diversity.' Page 1974. Arts, Humanities, and Museum Amendments of 
1985, Pub. L. No. 99-194. §1 10 (I) (G), 99 Sui. 1332. 1340 (1985) (Codified at 20 USC Section 960 (A) 1988). 

"'See text accompanying note 363, infra. 

"^Comment, supra note 256, at 1974. See also Comment, supra note 255, at 254. 

"'Panel Study Report, supra note 248, at 13. 
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diversity of opinion. "2« New Panels are reconstituted each year. Turnover rates vary from one-third 
to 100 percent per year.^' No panel member may serve for more than three consecutive years. ^ In 
all, about 800 people serve on NEA panels each year.^" 

The Program Office staff sends a "Panel book," containing a detailed summary of each 
application, the grant history of the applicant at NEA, and a brief discussion of any problems that the 
staff has identified, to each member of the relevant Panel or Panel section two to four weeks in 
advance of the Panel meeting.^ The Panel meets in Washington, D.C. for from one to six days, 
depending upon the workload.^ Panelists review all of the applications assigned to the Panel with 
reference to the review criteria published each year in the Program's guidelines. Some Panels divide 
up the work by assigning each member the role of "primary reviewer" for 10-15 applications. Other 
panels do not divide up the work. One NEA staffer related that his program tried the "primary 
reviewer" approach, but abandoned it after discovering that the remaining panelists did very little to 
familiarize themselves with the applications. 

Panel meetings are taped and the staff always takes notes. The object of the meeting is to make 
a reconmiendation to accept or reject each application and to recommend funding amounts for those 
that are recommended for acceptance. Although each Panel evolves its own procedures for carrying 
out these evaluations. Panel meetings typically progress in five phases: "(1) presentation of material, 
(2) discussion of applications, (3) formulation and recording of panel judgments, (4) determination of 
recommended grant amounts, and (5) final review and adjustment of recommendations."^ In some 
programs, panelists are provided with ballots broken down in accordance with the review criteria, 
and panelists are asked to grade each proposal on a scale of 1-10 or 1-100 for each review criterion. 
The agency staffer responsible for averaging the scores can identify any "outliers" in which vote 
spreads are very large and ask the Panel to discuss those applications in more detail. The Panel then 
proceeds down the list of applications in rank order and assigns a funding amount to each application 
until its monetary resources are exhausted. 

The Director or Assistant Director for the relevant Program attends the meetings to ensure that 
the results are determined solely on the basis of the published criteria, that conflicts of interest are 
dealt with appropriately, and that the deliberations are carried out in an atmosphere of fairness. The 
Program Directors can deal with cases of apparent bias subtly during the meetings, or they can raise 
their concerns privately with the NEA Chairman after the meeting. Program Directors attempt to 
avoid becoming involved in substantive discussions, because they want to avoid the perception that 
government employees are interfering with the peer review process. According to one Assistant 
Program Director: "It is vital that we are perceived to be dealing with the field in an objective 
fashion and that all recommendations be made by the panels." 

After the Panel meetings,^'' the Program Director and staff meet with the Deputy Chairman for 
Programs to review panel recommendations that were particularly controversial or raised special 

^®Panel Study Report, supra note 248, at 13. The Independent Conunission on the National Endowment for the Arts, supra note 232, 
at 27. ('Expert knowledge, dedication to the arif, aesthetic diversity, geographic distribution, ethnic and gender representation, ability to 
serve and to articulate issues in the field are all taken into consideration in this process.*). 

^'For sonte Programs, a 100 percent turnover rate is important for preserving the appearance as well as the reality of objectivity. For 
example, in the Visual Arts Program, the Panel considering applications for individual fellowships is completely replaced every year. 

^45 U.S.C. §959(c){6). 

^'^Facts about the Art Endowment. In 1989, nearly 800 persons served on the 103 panels that NEA convened. The Indepertdent 
Commission on the National Endowment for the Arts, supra note 232, at 25-26. 

^'^ost of the larger progranu use more than one Panel. For example, the Music Program uses thirteen Panels. Panel Study Report, 
supra note 248, at 1 1 . A few Panels serve more than one Program. Id. 

^^^The Independent Commission on the National Endowment for the Arts, supra note 282, at 30. 

^'^Panel Study Report, supra note 248, at 21; The Independent Commission on the National Endowment for the Aru, supra note 282, 
at 30. 

^^Because the Challenge Grants Program cuts across all substantive program areas, it conducts a second review by a 'super panel* 
made up of panelists from the 14 original screening panels to place the projects from all 14 panels in rank order and recommend funding 
levels. 
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policy concerns, any significant trends observed, and any issues that might cut across two or more 
Programs.^ The Chairman may ask to review particular files, and he occasionally convenes 
additional meetings with the staff to discuss particular applications or issues. When the staff or the 
Chairman believes that a panel has made a mistake, the panel can be called back to Washington for an 
additional meeting, but this happens only very rarely. The Chairman rarely reverses a panel 
outright. As a practical matter, the "advisory panels, through their recommendations, have come to 
be the determining element in the grant making process."^ 

The next step in the process is review by the National Council on the Arts. After being sued by 
two newspapers under the Federal Advisory Committee Act, the agency decided in 1990 to open all 
Council meetings to the public. Prior to the Council's quarterly meetings, the staff prepares a book 
for each Council member containing summaries of the applications that were recommended for 
approval and other materials regarding the panel discussions. These books are not made available to 
rejected applicants or to the general public. Lists of rejected applications are generally included in 
the books. Council members may review one or more of the applications if they desire, but they 
rarely see more than the staff-prepared summaries. The Council usually votes on the 
recommendations of a panel in a block and only very rarely singles out individual applications for 
separate votes."" 

After the Council meeting, the applications go to the Chairman for final action. The Chairman 
usually follows the Council recommendations, which in turn are usually based upon the Panel 
recommendations. The 1990 Amendments to the agency's statute prevent the Chairman from 
overruling the Council's recommendation not to fund an application."' On occasion, the staff will 
urge the Chairman to overrule the Council when the Council departs from the recommendation of the 
panel for reasons that the staff deems to be inappropriate."^ Although the Council is informed of the 
Chairman's action on each application, the Chairman is not required to provide any explanations. 
Some Chairmen are very careful to prepare detailed justifications couched in the language of the 
published criteria. Others provide little, if any, justification for rejecting Council recommendations. 
The Program officers then prepare acceptance or rejection letters for the applicants. Rejection letters 
are usually form letters containing only general information and the applicant's score, but 
occasionally they are more personalized. 

NEA's publicly available materials are vague as to how the agency fulfills its Privacy Act 
obligations. Disappointed applicants may request and receive "information about the panel review of 
their applications" by calling or writing the Program, but the agency's regulations and guidelines do 
not say whether the information available to applicants includes any minutes of Panel or Advisory 
Council meetings or any summaries of any individual peer reviews."' It rather clearly does not 
include a copy of the transcript of the tape recording of the meeting. The Endowment's regulations 
on information availability speak only to procedural issues,"* and agency practice varies widely. 
Some Program staffers are willing to discuss the contents of Panel meetings in some detail over the 
telephone with rejected applicants; others prepare summary statements from staff notes and meeting 
transcripts; and still others do not undertake to interpret the results but let the scores speak for 



^**Panel Study Repon, supra nou 248, at 23. For example, on one occasion, the Director of the Visual Arts Program disagreed with 
a panel recommendation that a choreography group be funded because the Director believed that the particular project did not come within 
the domain of visual arts. The Program Director there recommended disapproval of the proposal, and the Chairman agreed. 

^^^Fhe Independent Commission on the National Endowment for the Arts, supra note 232. 

"^d. at 30-31 (1990). 

"'20U.S.C. §955(0. 

"^For example, one member of the Council is firmly committed to the proposition that NEA should not fund criticism in any form, 
despite the fact that nothing in its charter prevents funding such projects. On one occasion, the staff urged the Chairman to overrule the 
Council's decision to reject two grants related to arts criticism that had been recommended by the respective panels. After the 1990 
Amendments to the agency's statute, the Chairman is no longer empowered to overrule a recommendation not to fund a particular project. 
See note 271, supra. 

"'45C.F.R. §1115. 

""45 C.F.R. pi. 1100. 
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themselves. Disappointed applicants may request that the General Counsel correct a record, 
including presumably errors on a written evaluation. If the General Counsel agrees with the program 
office that the request should be denied, the matter is referred to the Deputy Chairman for resolution. 
If the Deputy Chairman agrees, then a letter is sent to the applicant informing him or her of the 
decision and the reasons therefore. The applicant may then request an informal hearing before the 
Chairman or Assistant Chairman. ^^ 

Reaaing to charges that panelists were frequently participating in deliberations on applications 
from institutions that employed them. Congress in 1990 amended the agency's statute to prohibit 
individuals who are employees or agents of an organization with an application pending from serving 
on a panel considering that application."* The Arts Endowment has amended its "Standards of 
Condua for Council Members and Arts Endowment Panelists" to reflect this stricter conflict of 
interest requirement.^ The standards also address some additional aspects of bias in application 
review. The standards are "designed to avoid situations which may compromise, or appear to 
compromise, their objectivity while not preventing their continuing involvement in the arts."^ 

The standards contain a general direction to every Council member and panelist "to avoid any 
action which is, or could be interpreted as, a use of Council membership or panel service to further 
his or her own interests or those of an organization which he or she is affiliated."^ Council 
members may not submit an application on behalf of themselves or sign an application on behalf of 
an organization with which they are affiliated, and they may not participate in any way in a decision 
involving an application from an organization that employs them or with which they are affiliated. 
Panelists may not submit an application on behalf of themselves or through a fisc^ agent or as a 
collaborator or on behalf of an organization that employs them or for which they are an agent if the 
application will be evaluated by the subpanel (or Panel, if it is not divided into subpanel) on which 
they serve. 2» 

The regulations provide that Council members may not participate in the review of an 
application from an organization with which they are affiliated, and they must "avoid affecting or 
appearing to affect the decisionmaking process in any way."^' If Council members or organizations 
with which they are affiliated are recommended for funding by a panel, they may not panicipate in 
the review of the application. The Council member must physically leave the room during the 
consideration of such proposals. Panelists may not serve on a panel reviewing an application for an 
organization that employs them or for which they act as an agent. ^ All communications with the 
Arts Endowment concerning an application or grant must be carried out by personnel who are not 
Council members or panelists.^ 

Council members and panelists may receive remuneration for participation in funded activities, 
but only if the Council and the Endowment staff know the approximate amount of the remuneration 
prior to acting on the application. Of course, they may not participate in the consideration of such 
applications.** The propriety of receiving remuneration depends on "the nature of the organization. 



"*45C.F.R. §1115.5. 

"^0 U.S.C. S951 el acq. Prior to 1990, the sutute prohibited a panelist from submitting an application to his or her panel, but did 
not speak lo organizations with which the panelist was afTiliated. 

"^Memorandum to Members, National Council on the Arts and Aru Endowment Panelists from Amy Sabrin. General Counsel, on 
'Sundards of Conduct for Council Members and Aru Endowment Panelisu, dated September 18, 1991 (hereinafter cited as NEA 
Standards of Conduct). 

"*NEA Standards of Conduct supra note 277, at 2. 

"»ld. 

280Id. at3. 

*'ld. at 4. 

2«"Md. 

2*3ld.at3. 

^Id. at 4. 
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the amount of Endowment funding in relation to the total budget of the organization, and other 
relevant factors. "^^ 

Council members and panelists may not make use of confidential information acquired as a result 
of their service "in any manner which would advance their financial interests. "2«' This prohibition, 
however, is not intended to prevent Council members and panelists from supporting artistic 
endeavors with which they become acquainted during their service.^ 

To aid in the enforcement of these conflict of interest prohibitions, the Standards of Conduct 
require Council members and panelists to provide timely reports to the Endowment concerning their 
employment or interests (financial or otherwise) in organizations eligible to receive NEA money. 
Possible conflicts of interest are also discussed at the beginning of every Council meeting and Panel 
session. 

The Standards of Conduct speak directly to the touchy issue of animus. They provide that when 
a Council member or panelist or an organization with which he or she is associated is in the midst of 
an adversarial economic relationship with an applicant, the member or panelist must leave the 
meeting during which the application is being considered. The Standards of Conduct do not 
elaborate on the novel concept of adversarial economic relationship. For example, they do not 
address the question whether being the beneficiary of a competing application constitutes such a 
relationship. The Standards of Conduct are careful to provide that Council members and panelists 
may advocate general policies for adoption by the Council, even though the policies might have an 
adverse impact on competitors, so long as the policies would be implemented in a general way to all 
applicants.^ 

NEA has a formal appeals process whereby a program specialist or other agency staffer, but not 
a rejected applicant, can obtain a reconsideration of a rejected application. ^^^ Within 30 days 
following a decision not to fund, a Program specialist or other "authorizing official" in one of the 
programs may request an explanation for the decision not to fund from the relevant Program 
Director. The Program Director must then provide the requester with an explanation together with 
"the substance of the advisory panel review comments." The requester must be given an opportunity 
to "present his or her point of view. "^^ 

Beyond this strictly internal review for which the rejected applicant must have a "sponsor" 
within the agency, a rejected applicant may request reconsideration solely on the following 
procedural grounds: (1) the panel relied on criteria outside of those listed in the published guidelines; 
(2) the application was declined "based on influence on advisory panel [sic] of member(s) with 
undisclosed conflict of interest"; or (3) the application was declined "based on information provided 
to the advisory panel by staff or panelists that was materially inaccurate or incomplete at the time or 
review despite the fact that the applicant had provided the Endowment staff with accurate and 
complete information as part of the regular application process." This process may only be invoked, 
however, if a program specialist or other authorizing official has sought and received an explanation 
from the Program Director.^' After consulting with the Chairman, the appropriate Deputy Chairman 
must then reconsider the action and within 45 days provide a written summary of the results of his 



2«id. 

2**Id. at 9. 

^^^Id. al5. 

2**Id. 

^'^atioiul Foundation on the Arts and the Humanities, Federal Assistance; Reconsideration of Declined Applications, 61 Fed. Reg. 
131 18 (1983) [hereinafter cited as Reconsideration Regulations]. The statute does not explicitly provide for appeals from denials of grants. 
Conunent, The National Endowment for the Arts: A Search for an Equitable Grant Making Process, 74 Georgetown Law Journal 1491, 
1545 (1986). 

^^^econsideration Regulations, supra note 289, at 13118. 

2" Id. 
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reconsideration. The Deputy may request that surveys or site visits be conducted with respect to 
applicant organizations requesting reconsideration.^ 

From the foregoing description it is clear that while reconsideration is a possibility, it is not 
within the control of the rejected applicant. If the applicant does not have a sympathetic sponsor 
within the agency willing to "go to bat" for the project, the opportunity for reconsideration is 
effectively lost. The agency makes it clear, however, that a new application is welcome in the next 
funding cycle. 

D. The Process in Operation 

The extensive peer review-based system at NEA is intended to assume the principal burden of 
making qualitative judgments on the merits of proposals while at the same time assuring that "Federal 
assistance does not lead to government dominance or control in the sensitive area of aesthetics, 
artistic expression, and creativity."^ The concern for avoiding undue governmental influence on the 
content of artistic work distinguishes NEA from NIH, NSF, EPA and other scientific funding 
agencies. In the scientific agencies, there is a general reluctance to allow the government to tell a 
scientist how to practice science, but the funding agencies have a responsibility to steer research in 
directions relevant to their missions. In the arts, attempts to steer grantees in particular directions 
may be construed as government dominance of expression or creativity. Although there are always 
disputes at the margins between mavericks and adherents to the dominant paradigm, there are 
generally agreed upon criteria for "good science" in most fields, and the peer review committees are 
not at all unwilling to downgrade a proposal that does not manifest good science. The distinction 
between "good art" and "bad art" is much more elusive, and judgments about the quality of the 
subject matter of a proposal can easily be interpreted as attempts to steer artistic expression in 
particular directions. Past NEA Administrators have assiduously attempted to avoid invoking the 
specter of a "cultural ministry" through which the state determines what art and music the public 
should and should not enjoy. 

1 . Peer Review Under Fire 

Throughout most of its history, NEA's heavy reliance on peer review panels to guide 
discretionary dollars to promising artists and musicians has shielded the agency from attack by 
disappointed applicants and irritated taxpayers.^ As recently as 1981, a White House Task Force on 
the Arts and the Humanities, appointed by President Reagan and chaired by actor Charlton Heston 
praised NEA's peer review system as an "achievement" of "ongoing and widely accepted 
effectiveness.""^ In recent years, however, NEA" funding decisions have come under intense public 
scrutiny. Most of the controversy has involved the substantive outcome of the process-critics do not 
agree with the agency's funding decisions. Some of these criticisms, however, have been directed to 
the peer review process itself 

The Serrano /Mapplethorpe Controversy and the Helms AmendmentS'ince its inception NEA has 
been criticized for funding "frivolous" projects that, in the critics' opinions, wasted taxpayer money. 
Early in its history, NEA won Senator Proxmire's infamous "golden fleece" award for a $6000 grant 
to film the descent from an airplane of a four-mile-long piece of crepe paper. ^ But these criticisms 



^id. 

^^AiU Eitdowment Panels, lupra note 253. at 2. See Comment, supra note 256. at 1972 ('the drafters and early officials turn to 
professionalism • deference to the judgment of acknowledged art supporters and professional artists and organizations - to serve the goals 
of both depoliticization and excellence.*) 

"*Gracc Glueck. A Federal Benefactor of the Axu Come of Age, New York Times. Nov. 10. 1985, sec. 2, p. 1, col. 1 (recounting 
NEA's successful defense against attacks by a Texas Congressman for supporting allegedly pornographic poetry); McLellan. supra note 8 
(recounting several failed external attacks on NEA in response to individual funding decisions and quoting the Chairman of NEA to the 
effect that the peer review system protects him from political pressure). 

"*Report Alleges NEA Rivalry, Washington Post, August 3. 1981, CI. 

"^banquette, supra note 226. 
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were generally considered par for the course for an agency that sometimes took risks in the pursuit of 
creativity. As with the scientific agencies, which have also received their share of such awards, these 
attacks were sufficiently few and far between that they posed no real threat to the agency's routine 
operations. 

The recent controversies over NEA's funding of allegedly obscene and indecent art have, by 
contrast, shaken the agency to the core. In 1988, a leaderless NEA provided funds to the Institution 
of Contemporary Art of Philadelphia to display an exhibit of the photographs of the recently deceased 
artist Robert Mapplethorpe.^ Although the vast majority of the photographs were entirely 
unobjectionable, one part of the exhibit was a series of small photographs, entitled the "x, y, z 
series," that contained graphic representations of homosexual sexual conduct and sado-masochism.'" 
At roughly the same time, NEA also funded an exhibition at the Southeastern Center for 
Contemporary Art in Winston-Salem, North Carolina of photographs by Andres Serrano that 
included the soon-to-be-infamous "Piss Christ" photograph that featured a crucifix immersed in a jar 
of the photographer's urine.^ The predictable public outcry against the use of taxpayers' money to 
support such projects starkly raised the issue of NEA's political accountability and especially the 
accountability of its peer review panels.'" 

At the behest of Senator Jesse Helms, the Senate Appropriations Committee in January 1989 
voted to prevent NEA from providing any appropriated funds to the offending institutions for a 
period of five years. '"' The full Senate, without debate, substituted an appropriations rider, later 
known as the "Helms Amendment," that prohibited NEA from funding art that would "promote, 
disseminate, or produce obscene or indecent materials, including but not limited to depictions of 
sado-masochism, homoeroticism, the exploitation of children, or individuals engaged in sex acts; or 
material which denigrates the objects or beliefs of the adherents of a particular religion or non- 
religion."*^ This precipitated a pitched battle between arts supporters and NEA critics in the Senate 
and House, the ultimate outcome of which was a watered down version of the Helms Amendment that 
prevented NEA from funding art that "may be considered obscene . . . and which, when taken as a 



^'^See fno author]. Chronology of a Controversy, The Christian Science Monitor, July 31, 1990, p. 10. 

For a more detailed description of the Mapplethorpe controversy, see Owen M. Fiss, *Suie Activism and Slate Censorship,* 100 
Yale L. J. 2087 (1991) (characterizing the Mapplethorpe exhibit as "a considerable achievement,* presenting 'an aesthetic vision that is 
original and in many respecu stunning*); Comment, supra note 255. 

When the Mapplethorpe exhibit was moved lo the Contemporary Art Center in Cincinnati, the Center was indicted by a local grand 
jury. A jury later found the defendants not guihy. See Comment, supra note 255, at 241-43. 

^"For a more detailed description of Serrano's work, see Jeff Lyon, Point Man, Chicago Tribune, Jan. 27, 1991 , Sunday Magazine, 
p. 12. Soon to follow was Serrano's 'Piss Pope' and similar denouncements of prominent authority figures. Apparently, Serrano's 
particular talent is the art of bodily fluids. According to Tony Jones, the President of Chicago's School of Art Institute, Serrano 'is 
commenting on his relationship with the Catholic church and Catholic teaching, and the fluids are lo capture the idea of trans- 
substantiation.* Lyon, supra. 

'"'^Congressman Fred Grandy. a former television actor and a supporter of NEA observed: 'Its an argument of perception, and you 
have a lot of people squandering money. I've been telling the artists that they need to come with something that gives the impression of 
accountability.* Richard Elemstein, The Outlook on Arts: Grants, New York Times, June 26, 1990, sec. C, p. 13, col. 4. 

Barbara Gamarekian, Senate Panel Asks Ban on Grants to Two Arts Groups, New York Times, Jan. 26, 1989, sec. C, p. 19, col. 
1 . NEA's acting director protested the action, calling it *a dangerous precedent which goes against the long and widely respected system 
of Federal Support for the Arts through a competitive peer-review process.' Id. 

Ted Potter, the Executive Director for the Southeastern Center for Contemporary An, complained that his institution was 'being 
punished for doing exactly what are supposed to do: challenge the public to see, think, and discuss critical issues of our culture and 
society. The role of contemporary arts and the institutions that foster it is to take risks and confront the established dogmas with innovative 
insights and to offer challenging exhibitions and experiences.' supra at sec. C, p. 19, col. 1 . 

'"Comment, supra note 255, at 235-39. (describing the process oferucting the Helms Amendment); Michael Oreskes, Senate Votes 
lo Bar U.S. Support of *Obscene or Indecent* Artwork, New York Times. July 27, 1989, sec. A. p. 1, col. 2. 

In some respects, the amendment harked back to an earlier failed attempt by Congressman Biaggi lo prohibiting NEA funding of any 
productions containing 'any ethnic or racially offensive material' af\er the lialian-Ameni:an community was insulted by an NEA funded 
modernization of the opera Rigoleao. See Comment, supra note 289. 
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whole, does not have serious literary, artistic, political or scientific value."*'' Reacting to the strong 
political statement contained in the Helms Amendment, NEA Chairman John Frohnmeyer 
administratively added to grant application forms a "certification" through which grant recipients 
agreed not to produce "obscene" art with NEA money. ^ 

Senator Helms' efforts inspired other critics of NEA to air their complaints about conflict of 
interest on the part of members of NEA peer review panels. A study conducted by the Washington 
Times purported to show that at least 130 of approximately 730 peer panelists who assisted NEA in 
1988 sat on panels that recommended grants to their own organizations. The survey also reported 
that "[i]n several cases, . . . grant reviewers were named as principal beneficiary of federal grant 
awards by their own panels."*" An NEA attorney admitted that it was not unusual for a panelist to 
sit on a panel that reviewed an application from an institution employing that panelist. NEA 
regulations at the time allowed this, so long as the panelist left the room during the deliberations on 
his or her institution's application. One NEA critic was unimpressed with this solution to the conflict 
of interest problem, arguing that: 

It is simply insufficient to say 'I'll leave the room (for debate on my grant)' .... 
"Your buddies are back there. The NEA knows before they set up the panels 
whether a person ... is putting a grant proposal. This looks to me like more than 
seeming impropriety. This looks like impropriety itself** 

Another NEA critic observed: 

The very least that can be said of the system is that it was very much a buddy system. 
You give me a grant, I give you a grant, that sort of thing. Among professionals, 
they joke a great deal among themselves about how it works.*'' 

Supporters of publicly funded arts and music responded with a vigorous defense of 
the peer review process** and launched a full-scale offensive against broad-brushed 
government "censorship" that the Helms Amendment allegedly represented.** Some 
scholars took the position that such content-based restrictions on government-funded 



"''Department of the Interior and Related Agencies Appropriation Act of 1990 Bup. L. No. 101-121. Tit. HI, _. 304. 1989 US Code 
Cong. •Admin. News (103 Sui.) 701, 741-42. 

**See Owen M. Fiss, supra note 298. Comment, supra note 255, at 239. 

*^George Archibald, NEA Review Panelists Run Grant-Giving 'Buddy System, Washington Times, Aug. 3, 1990, p. Al . 

** Allan Paracini, Conflict of Interest Issue in NEA Grants? Los Angeles Times, July 27, 1990, pt. F, p. I, col. 5 (quoting Professor 
John Agresto, of Sanu Fe College). 

*' Archibald, supra note 305 (quoting Mr. Hilton Kramer). Kramer also suggested that the peer-review system was "a corrupt way lo 
make decisions about (he best way to spend the agency' money,* and that it had 'degenerated into a sort of political club in which certain 
insidera knew well in advance where the money would be going.* Jonathan Yardley, In Conclusion, Reason But No Raison D'Etrc, 
Washington Post, Sept. 17, 1990, B2. 

in a letter to the editor of the New York Times, Harvey Lichlensiein, President and Executive Director of the Brooklyn Academy of 
Music, gave the following defense of peer review at NEA: 

[T)he [peer review] system has worked superbly for almost 25 years of grant making by NEA .... I have found that 
the process of considering grant requests by the NEA panels and program directors impressively fair, (and] they have 
functioned with integrity and generosity of spirit. There has never been a hint of financial scandal or mismanagement, 
and the range of grants has cut through every ideological boundary. 

Harvey Lichtenslein, Where Are We Heading? New York Times, July 30. 1989. sec. 2, p. 33. col. 5. See also Editorial. The Helms 
Process, New York Times, July 28, 1989, sec. A, p. 26, col. 1 (*So far. Congress has wisely relied on a peer-review process by which 
membere of the arts community pass on grant applications in their respective fields. Now Senator Helms &. Co. insist on what could be 
called the Congress-as-connoisseur process.*). 

*'john Farrell and Max Benavidez, Let the Arts be Unfettered by Politics of the Moment. Los Angeles Times, Aug. 7, 1989, pt. 2, p. 
5, col. 1 (characterizing the Helms Amendment as 'the most serious and radical assault on freedom of expression to occur in this country 
since the days of Joe McCarthy and 'black list."); Editorial^ The Helms Process, supra note 308 (arguing that the Helms Amendment 
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art was unconstitutional under a variation of the doctrine of "unconstitutional 
conditions.""" Even those who did not contend that restrictions on government 
funding amounted to censorship complained that the Helms Amendment, which had 
been enacted without congressional debate,"' represented bad public policy. "^ 

Chairman Frohnmeyer's "certification" requirement, which was likened to a loyalty oath, 
outraged many grantees,^'' precipitated discord within the agency, and generated a rare 
recommendation from the National Advisory Council on the Arts that it be revoked."* It was 
eliminated after a federal district court, on January 9, 1991, held that it was unconstitutionally vague 
and had a unconstitutional chilling effect on the first amendment rights of grant recipients."^ 

The "NEA Fowr "-Until fairly recently, NEA Chairpersons only very rarely overturned the 
recommendations of the review panels. Livingston Biddle, NEA Chairman during the Carter 
Administration, could not recall a single instance in which he overturned a panel recommendation 
that a project be ftmded."' President' Reagan's first NEA Chairman, Frank Hodsell, adopted a 
somewhat more proactive role, rejecting twenty applications recommended by review panels within a 
two year period."'' Nearly all of these rejections were uncontroversial, and despite some early 
concerns, overtly political considerations were apparently absent from NEA funding decisions."' 
The controversy over the Mapplethorpe/Serrano exhibits and the Helms Amendment, however, 
subjected Hodsell 's successors' actions to more intense public scrutiny. 

President Bush's first appointee to head NEA was John Frohnmayer, a former college football 
star, amateur opera singer, and trial lawyer from Oregon who had little experience in managing large 
agencies or public controversy."' Almost immediately upon assuming office, he created a stir when 
he unilaterally withdrew NEA funding for an exhibit entitled "Witnesses" that focused upon the 



'would plunge one aesthetic question after another into the boiling bath of politics,* a prospect that was 'unlikely to be good for politics* 
and would 'aurely be faul to art*). 

^"^ee Fias, supra note 298, at 2096-2102. Comment, supra note 256. Rorie Sherman, NEA Grant Provisos Attacked in Couru, 
National Law Journal, Nov. 5, 1990, p. 12 (quoting Professor Nan Hunter). 

Editorial, More on Arts Funding, Washington Post, July 30, 1989, C6 (*The Helms amendment was proposed to a near-empiy 
chamber with little or no advanced warning, and with no exanunation of its implications for the existing peer-review system of awarding 
grants.*). 

"^ee, e.g., Matthew Gilbert, Writers Blocked? New Rules for Grants Met With Trepidation. Boston Globe, Mar. 7, 1990, p. 1; 
Howard Reich, The Arts Under Assault: Why the NEA is at the Center of a Fierce Morality Play. Chicago Tribune, June 24, 1990, p. 4. 
Interestingly, none of the recipients of NEA grants inunediately upon enactment of the Helms Amendment declined the funds, probably 
because all of them believed that their work at the very least had redeenung artistic or literary value. Sec William O'Rourke, Protesting 
NEA: National Endowment for the Arts and Jesse Helms' Obscenity Crusade, The Nation, June 25,1 990, vol. 250, no. 25, p. 880 ('All 
my depictions of sado-masochism and homo-eroticism have literary value, always have and always will.') 

"^See Dawn Shirley, Dramatist Take Sides in NEA Fight, Los Angeles Times, May 17. 1990. pt. F, p. 7, col. 3 (quoting playwright 
TerreiKe McNally as saying: "I think it's better to drive a cab or wail tables than sign such a pledge.*); Matthew Gilbert, Writers 
Blocked? New Rules for Granu Met With Trepidation, Boston Globe, Mar. 7, 1990, p. 1. The controveny over the certification 
requirement inspired the creation of a group, headed by author Larry McMurtry, the primary goal of which was 'unfettered funding for 
the art£.° Id. 

"*The Independent Commission on the National Endowment for the Arts, supra nou 232, at 88. Comment, supra note 255, at 239- 
40. Kim Masters, Arts Panel Urges End to Grant 'Pledge,* Washington Post, Aug. 4, 1990, p. Gl . 

"*BelU Lewitzky Dance Foundation v. Frc^nmayer. 954 F. Supp. 774 (CD. Cal. 1991). NEA eliminated the certification 
requirement as part of the settlement of a second lawsuit pending in New York. Fiss, supra note 298. at 2095 n. 25. 

"Robert Pear, Reagan's Arts Chairman Brings Subtle Changes to the Endowment. New York Times. April 10, 1983, sec. 2, p. 1., 
col. 1. Mr. Biddle took the strong position that the peer-review panels were a *bulwark of strength protecting the endowment from the 
whims and fancy of one person who is a federal bureaucrat.* 

^'^Id. See also Grace Glueck, supra note 294, at sec. 2, p. I, col. I. 

"^Grace Glueck. supra note 294, at sec. 2, p. 1, col. 1 (noting that NEA during the Reagan Administration funded painter Peter Saul, 
who was known for his vitriolic portrayals of President Reagan). 

^"Rorie Shennan, Calm Presence in the Middle of Arts Battle, National Law Journal, July 2, 1990, p. 8. 
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AIDS crisis."" At one point Frohnmayer explained that he toolc the action because the exhibit 
contained a letter from a dying AIDS victim that was harshly critical of Senator Helms, 
Representative Dannemeyer, and John Cardinal O'Connor, the Roman Catholic Archbishop of New 
York, Mr. Frohnmayer said that he took the unusual action of withdrawing funding because the 
exhibit was "political.""' When asked how he was able to distinguish between "political" and 
"apolitical" art, Mr. Frohnmayer explained: "If you came to us and said, 'I want to create a political 
polemic,' we would not ftind Uiat. If you, on the other hand, said: 'I want to paint a "Guernica"' we 
would fund that. And the difference is we make our decision on the artistic content. ""^ While it is 
easy to see how this subtle distinction was lost on the peer review panel that originally recommended 
that the project be funded, it is harder to identify the criteria by which the agency's political 
appointees could draw such distinctions in the future. To make matters worse, Mr. Frohnmayer 
allowed that "[t]he word "political" means something quite different in Portland, Oregon, than in 
Washington. "323 

Perhaps unintentionally, Mr. Frohnmayer put his fmger on one of the most serious problems 
with the application of the peer review model to the arts. Much art is in fact "political" when it 
makes statements, subtle or otherwise, about the allocation of political and economic power in 
society.'^* Politics is an important and inevitable pan of the culture that surrounds and is reflected by 
literature, the visual arts and music. Art can no more be divorced from politics than it can be 
divorced from any other important aspect of contemporary life. The concept of "political" cannot 
adequately distinguish a polemic against Jesse Helms from a "Guernica," a bust of Lenin or a 
painting of Washington Crossing the Delaware. It can, however, be manipulated to steer art and 
culture into particular political directions. And this potential for subjecting government funded art to 
a "political correctness" test is precisely why Congress and early NEA Chairmen created the peer 
panel process to make the initial funding recommendations.'^ 

If political art is not necessarily "bad" art, however, it is unclear whether the concept of "artistic 
excellence" provides criteria for distinguishing the good from the bad that are any more objective and 
any less subject to political manipulation. Does that concept contain constraining criteria capable of 
preventing the professionals on the peer review panels from advancing their own hidden political 
agendas dirough their funding recommendations? Can the peer review process itself be arranged so 
as to provide checks against such abuse? If not, must a society that elects to devote significant public 
resources to art expect more-or-less continuous struggles for political control over this powerful form 
of public expression? 

Questions like these were at the center of the intense public debate that resulted from 
Frohnmeyer's decision on June 29, 1990 to overturn a peer review panel's recommendation that the 
agency fiind projects by performance artists John Fleck, Holly Hughes, Karen Finley, and Tim 
Miller. Although all of the rejected artists had received NEA funding in the past,'* none of them 
could be considered mainstream. Ms. Finley became famous for pouring jello in her bra and 



"°Grace Glueck, Border Slurmish: Art and Politics. New York Times, Nov. 19. 1989, sec. 2, p. 1. col. 2; John Robinson, 
Frohnmayer Weathers Storm at NEA, Boston Globe, Nov. 18, 1989, p. 9; Editorial, Mr. Frohnmaycr's Fumble, New York Times, Nov. 
17, 1989, ace. A. p. 38, col. 1. 

"'Glueck, supra note 320. Frohnmayer explained that there had been an 'erosion of the artistic vision* of this show between the lime 
the grant was awarded and its production. Id. 



322, 
323, 



Id. 



'Id. 

"*Paul Mattick, Jr., Arts and the Sute, 251 The Nation 348 (Oct. 1. 1990) (The problem is that art has been politicized, the 
existence of state funding shows thai the generally political hidden side of the arts has existed all along. The struggle over the NEA is a 
struggle for control of this political side.*). 

^^See Fiss, supra nou 298, at 2093-96, Conunent, supra note 293. 

'^aula Span and Carla Hall, Rejected! Portraiu of Performance Artists The NEA Did Not Sec Fit To Fund, Washington Post 
National Weekly Edition, July 16-22, 1990, p. 10. 
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smearing chocolate on her naked body while on stage to symbolize "women being treated like dirt."'^ 
Ms. Hughes' satiric performances, which she described as "storytelling," "celebrate[ed] her lesbian 
sensibilities."'^ Tim Miller was a gay performance artist who also performed in the nude.'^ In one 
NEA-funded performance, John Fleck urinated on the stage,'* and in another he simulated his male 
and female halves making love."' 

Although sufficient funds were available in the program to fund the applications, a worried 
Frohnmeyer telephoned the members of the panel privately to ascertain the reasons for their 
recommendations. "2 Frohnmeyer told a group of arts supporters in Seattle that "certain political 
realities" made it unlikely that he would decide to fund all of them."' The National Advisory 
Council on the Arts, in a rare move, put off a final recommendation until its August 1990 meeting. 
Instead of waiting for the results of that meeting, Frohnmeyer polled its members by telephone and 
determined that a majority of the Council opposed funding the proposals. Transcripts and notes of 
these telephone conversations released in subsequent litigation revealed that concerns for the 
"political" consequences of making the awards permeated the discussions."* The fact that this 
decision came within a month of a strong statement by the White House press secretary against 
NEA's support of "obscene" art"^ suggested to some ^at Frohnmeyer 's decision may have been 
affected by pressure from the White House. 

The rejected artists did not suffer their defeat quietly. Mr. Miller called Frohnmayer a "sleaze- 
ball appointee" of "King George Bush.""* Ms. Hughes wrote in the New York Times that: "[t]he 
overturning of these grants represent Mr. Frohnmayer 's and President Bush's attempt to appease the 



'^Barbara Gamarekian, Head Of The NEA Is Said To Accept 'Political Realities," New York Times. June 29, 1990. Bl. col. 5 
(quoting Karen Finley). Ms. Finley maintains that '[t]here is nothing sexually exciting about my work; if there were I would be doing 
burlesque shows and centerfolds and making a lot of money.* Culture critic Robert Friedman agreed: 'Watching her smear her naked 
body with chocolate, then cover herself with alf alfa sprouts and tinsel, is about as sexual arousing as watching Julia Child bake a cake.* 
Robert Friedman, From Yanu to Chocolate, Newsday, Aug. 7, 1990, p. 54. Other observers, however, *have found less comforuble 
parallels between Finley's work and conventional pornography.* Hilary DeVries, All The Rage: Karen Finley Has Become In The 
Struggle Over Public Art Support, Los Angeles Times, Oct. 21, 1990, p. 3. Finley acknowledged that her work was often overtly 
political: 'It's about social issues they don't want to hear about. This is their last chance at trying to mainuin the power structure of the 
straight white male.' Paula Span and Carta Hall, Rejected!, supra note 326. 

'^Paula Span and Carla Hall, Rejected!, supra note 326. Art critic Cathy Curtis described Hughes' work as follows: 

Holly Hughes' work involves a poetic transformation of the everyday activities of women into gestures that reveal the 
elemental sexual nature shared by all of humanity. She strips away social veneer to uncover the primal essence of her 
characters. This is art that treats the hunfian condition in a metaphorical way, with lyricism, warmth, and humor. 

Cathy Curtis, Defending the Avant-Garde From the Hysterical Helms of Conservatism, Los Angeles Times, July 30, 1990, pt. F, p. 3, 
col. 1. 

'^^r. Miller esplained: 'In a work called 'Some Golden Sutes,' there is one section right at the beginning of the piece where 1 tear 
my clothes ofT and I'm basically kind of thrown around on a pile of leaves. It's a piece I made about the reality of my hometown, 
Whittier, being destroyed in the 1987 earthquake.* Paula Span and Carla Hall, Rejected!, supra note 326. 

"'^eck explained: 'Whoopie doopie. Taken out of context is sounds like sensationalism. Bui there was some social relevance 
attached. It really wasn't disgusting or rude or crude. I don't think anybody was offended by it. In fact, I think if you look back it's 
nothing new. It's been done before.* Paula Span and Carla Hall, Rejected!, supra note 326. 

"'Paula Span and Carla Hall, Rejected!, supra note 326. 

"kJeorge Archibald, Speculation in Arts World Has Frohnmayer Quining, Washington Times, July 2, 1990, pt. A, p. A3. 

"'Barbara Gamarekian, supra note 327. 

"^ee U.S. Documents Said to Show Endowment Bowed to Pressure, New York Times, September 18, 1991, Al, Col. 5. The 
transcripts contain one very revealing interchange that says a lot about the tension between professional values and democratic 
accountability. At one point, Frohmneyer queried the panelists: 'Let me ask the very crass and difficult political question, what am I 
going to aay when one of our critics comes in . . . and says 'Gees, they funded a guy who whizzes on stage?" One of the panelists 
responded: 'Who knows? Who cares? They're good.* Kim Masters, Politics of Arts Grants Questioned, Washington Post, September 18, 
1991. Bl. 

"^Comment, supra note 255, at 249, (quoting Bedard & Archibald, 'Bush Has a Change of Art,* Washington Times, June 13, 1990, 
at Al). 

"* Allan Paracini, Cal Arts May Turn Down NEA Money, Los Angeles Times, July 3. 1990, pt. F, p. 1, col. 6. 
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homophobic, misogynist and racist agenda of Senator Jesse Helms and company.""' Some artists 
who received funding vowed t^ return their grants to NEA or give portions of them to the rejected 
artists.''* Members of the The. • er Arts peer panel complained bitterly in public about the Chairman's 
rejection of the panel's recommendations,"' and the panel refused to make any more 
recommendations.'*' Even the mayors of Los Angeles, Seattle and New York protested the 
decision,**' In his defense, Frohnmayer explained: "I think we cannot look strictly at artistic 
excellence in a vacuum, but we have to look at it has how it is going to play with the audience that 
we're charged with serving-which is the people."^ 



•342 



The Independent Commission Report-One of the results of the 1989 Mapplethorpe controversy 
was legislation creating an "Independent Commission" to review NEA's grant making procedures, 
including the peer review process, and to "consider whether the standard for publicly funded art 
should be different from the standard for privately funded art."*^ President Bush appointed to the 
Commission a group of prominent individuals from both sides of the great NEA debate. Within a 
relatively short time, the Commission produced a unanimous report that adopted a consistent middle- 
of-the-road posture toward both of the issues that it was assigned to investigate. *** 

On the question whether publicly funded art should be judged by different standards than 
privately funded art, the Report concluded: 

Publicly funded art must be measured, like privately funded art, against standards of 
artistic excellence. Publicly funded art must be judged by other standards as well. . . 
. It should be chosen through a process that is accountable and free of conflicts of 
interest. It should be selective with an awareness of the geographic and cultural 
diversity of the United States and with respect for the differing beliefs and values of 
the American people. . . . Insuring the freedom of expression necessary to nourish 
the arts while bearing in mind limits of public understanding and tolerance requires 
unusual wisdom, prudence, and most of all, common sense. "'^^ 

With respect to NEA's decisionmaking procedures, the Commission noted that Congress had 
never meant for the National Council and the peer review panels to be the fmal decisionmakers, and 
it stressed that only the Chairman was accountable to the public through the political process.*** The 
Commission went further to recommend that the Chairperson "be given more authority and more 



"^Holly Hughes and Richard EJovich. Homophobia at ihc NEA, New York Times. July 28. 1990. A15, col. 3. 

"^Paracini, lupra note 336; William H. Honan, Anti-Obscenity Pledge for Grants is Modined. New York Times, July 2, 1990. A13, 
col. 5. 

'"Joyce Price, NEA's Rejection of Four Grants Riles Advisory Panel. Washington Times, July 9. 1990, p. A3. Phillip An>oult, 
Chairman of the Solo Perfomunce Fellowship Panel said: *I am very concerned about the four artists, and I have an equal concern about 
the peer-review panel process and how it was dealt with in a very unusual way.* Id. The panel issued a joint statement deploring the 
action 'and the political manipulations which prompted it.* Allan Paracini, Mayor Blasts NEA Awards S3 Million In City Funds, Los 
Angeles TinKS, July 12, 1990, pi. F, p. 1, col. 2. Jacob Neusner, a member of the Advisory Council, responded: "This is gobbildy gook 
from aelf-righteous people who make decisions for their own purposes, not as a result of values and sensitivities that nutter to others.* 
Price, supra. 

'^''Melinda Bargreen, The Embattled Ans Agency Finds Itself Surrounded On All Sides As Its Reauthorization Vote Draws Near: 
Targeting The NEA. The Seattle Times. July 22. 1990, LI . 

'^'Paracinj, supra nou 339. 

**^William H. Honan, Aru Agency Says Public Is Losing Confidence, New York Times. July 3. 1990, Bl. col. 6. As is of\en the 
case in such matlera. the controversy apparently attracted much greater interest in the performances of the NEA four. Not long afler the 
controveny broke out, Karen Finley performed to sold-out houses at the Lincoln Center. Hilary DeVries. All The Rage Karen Finley 
Has Become In The Struggle Over Public Art Support, Los Angeles Times, Oct. 21, 1990. p. 3. 

'^'The Independent Commission on the National Endowment for the Arts, supra note 232. at I . 

'**The IxK-.-pendent Commission on the National Endowment for the Arts, supra note 232 

'*5id.at2-3. 

'**Id.at 16. 
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choices."*'^ Noting that no other granting agency gave peer review panels such discretion over both 
whether to award grants and the amounts of those grants,*^ the Commission suggested that the panels 
and the Advisory Council "recommend more grants than funds available for them, thereby giving the 
Chairperson a genuine choice in awarding grants."^' 

The Commission also expressed concerns about conflict of interest among panel members, and it 
recommended that NEA "insure that panelists do not serve as members of panels that consider 
applications for grants from institutions with which they are affiliated."'* At the same time, the 
Commission urged the agency to "insure that the membership of panels represents a variety of 
aesthetic and philosophical views.""' The Commission noted that the potential for bias could be 
reduced by commissioning multiple panels in a single discipline, a solution that "would not only 
lessen the possibility of conflicts of interest but would reduce the number of applications before a 
single panel. ""2 In addition, the pool of potential panelists should be expanded to include "people 
who do not earn their living in the arts," such as "art critics, collectors, educators, trustees or others 
with an aesthetic knowledge of a particular discipline and a view of the world outside the arts."'" 
Finally, the Commission recommended that panelists' terms be limited to three years."* 

Although stressing throughout its report the need to maintain accountability, the Commission 
recommended that panel meetings remain closed to the public.'" The Commission listed several 
reasons for this position: 

First, panelists are likely to be more candid if they are discussing, behind closed 
doors, the work of a colleague or prominent competitor. Candor would also be 
compromised if an applicant were sitting in the room. Second, open panels are an 
invitation to pressure groups and lobbyists, seeking to influence the decision of 
panelists. Third, a frank discussion of a particular proposal could damage the 
reputation of an artist or institution. Finally, and of particular importance, public 
knowledge of a panel's deliberations prior to the consideration of an application by 
the National Council and the Chairperson could act as a restraint on their exercise of 
discretion.'^ 

At the same time, the Commission recommended that the agency maintain careful records of 
panel deliberations and make them available to rejected applicants if requested so that the applicants 
would "have access to the reasons their applications were accepted or rejected.""'' 

The 1990 Amendments-Tho, Helms Amendment also appeared in the agency's 1990 
appropriations act, but an attempt to included it in the 1991 appropriation died in conference 
committee.'^ Helms and his allies in Congress were not, however, willing to rely entirely upon 
appropriations riders to ensure that the agency adhered to their policies. When the agency's statute 
came up for reauthorization in 1990, Senator Helms and others attempted to write the anti-obscenity 



'^^Id. at 65. The Commission rejected a suggestion by Stephen Weile, an official at the Hirshom Museum and Sculpture Garden, that 
the agency's statute be annended to take away the Chairman's individual veto power and to force the Chairman to approve or reject all of a 
panel's recommendations en masse. Allan Paracini, NEA Panel Seeks Consensus as Time Pressure Builds, Los Angeles Times, Aug. 2, 
1990, pt. F. p. l.col. 5. 

'^^The Independent Commission on the National Endowment for the Arts, supra note 232 at 72. 

'^d. «t65. 

'»ld. 

"'Id. at 66. 

"^Id. at 74. The Commission also suggested that NEA consider awarding multi-year grants. Id. 

"'Id. at 74. 

"*Id. 

"^Id. at 76. 

'»Id. 

'"id. at 77. 

'^Comment, supra note 255, at 240 n. 48. 
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language into the agency's authorizing legislation. Supporters of the arts community in Congress 
struggled to limit the constraints that the statute placed on the agency. The result was a compromise. 
Relying heavily upon the Independent Commission's report, ^^ Congress declined to write a broad 
prohibition against funding "indecent" projects into the statute. Instead, the amendments required 
any recipient of a grant later determined to be "obscene" by a court to refund the grant to NEA. In 
addition the requirements for awarding grants were changed somewhat from a preference for projects 
that "foster excellence," and reflect "significant, literary, scholarly, cultural, or artistic merit" to an 
injunction that awards be based "solely on ... artistic excellence and artistic merit, "^ taking into 
consideration "general standards of decency and respect for the diverse beliefs and values of the 
American public."'" 

The amendments also addressed the grant awarding process. Reacting to charges that panelists 
sometimes suffered from conflicts-of-interest,'<° the amendments required NEA to issue regulations 
and establish procedures to ensure that "all panels include representation of lay individuals who are 
knowledgeable about the arts but who are not engaged in the arts as a profession and are not 
members of either artists' organizations or arts organizations."'® In addition, the membership of 
each panel had to "change substantially from year to year" with no member serving more than three 
consecutive years.'** The agency was also required to ensure that "an individual who has a pending 
application for financial assistance ... or who is an employee or agency of an organization with a 
pending application, does not serve as a member of any panel before which such application is 
pending."'" As related above, NEA has recently promulgated regulations implementing these 
changes. Having changed the process of selecting panelists. Congress went on to grant them 
somewhat more autonomy by providing that the Chairman of NEA may not approve an application 
with respect to which the National Advisory Council on the Arts has given a negative 
recommendation . '" 

Post-1990 Prflcr/c^-Although many in the arts community were alarmed by the 1990 
amendments,"" Chairman Frohnmeyer assured them that he would not become a "decency czar."'" 



^lie Independent Commission on the National Endowment for the Arts, supra note 232. at 89 (Commission recommends 'against 
legislative changes to impose specific restriciions on the content of works of an supported by the endowment*). 

'*°45 U.S.C. 8959(c). 

'*'45 U.S.C. §954(d)(l). Professor Fiss fears that this reference to 'general standards of decency' allows the NEA Chairman too 
much discretion to refuse grants that are not obscene, but are nevertheless controversial. Fiss. supra note 298, at 2094-95 (The 
chairperson is freed from the Miller sUndards and is able to deny fiinding to a project like Mapplethorpe's, even though it is not within the 
constitutional definition of obscenity and thus not amenable to criminal prosecution.') 

'^^See text accompanying notes 305-312, supra. 

'*'45 U.S.C. §959(c)(2). 

'*<45 U.S.C. §959(c)(6). 

'"45 U.S.C. §959(c). 

'**45 U.S.C. §955(0. I^espitc this change. Professor Fiss argues that the net effect of the 1990 amendmenu is to give the NEA 
Chairman more power vis-a-vis the panels. Fiss, supra note 298, at 2094. 

'*'See William Honan, Symposium Finding Fault With New Aru-Grani Law. New York Times, Nov. 10, 1990. A14. col. 1 
(reporting of a convocation of arts administrators, public officials and attorneys convened by the National Assembly to evaluate the 1990 
amendments). Joseph Papp, the Producer of the New York Shakespeare Festival found the amendments 'unaccepuble, reprehensible and 
insidious,* and he therefore rejected NEA grants worth S323,000. Gerald Frasier. Papp Rejects $323,000 in Endowment Grants, New 
York Times. Nov. 2. 1990, B3. col. 1 

'^William H. Honan, Aru Council Turns I>own Decency Rules for Advisors. New York Times. Dec. 15. 1990, A 15, col. 3. I>uring 
the time that Congress was seriously considering more restrictive versions of the 1990 amendments, Frohnmeyer had accepted the 
recommendation to the National Council that he reject the recommendation of the Panel on Special Museum Exhibitions that NEA fund an 
exhibition of the works of controversial artist Mike Kelley. No author, Frohnmayer Rejects Grant for Boston Show. New York Times, 
Oct. 23. 1990, C16, col. 1. According to the Director of Boston's Institute for Contemporary An, the location of the proposed exhibiiion: 
Mr. Keiley's work 'deals with serious and critical issues within our cultural, some of which deal wjih the pathology of our times, some of 
which deal with the body and references to sexuality and sociology, some of which deal with nationality and nationalism, some of which 
deal with madness and panicular forms of modem schizophrenic behavior.* (No author), NEAs Chief Rejects Grant to Boston Museum, 
Los Angeles Times, Oct. 22, 1990, pi. P, p. 10, col. 3. The Director speculated that The only reason this grant was turned down after 
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As if to prove his point, he quickly approved grants to Karen Finley and Holly Hughes, two of the 
"NEA four," thereby invoking the wrath of the anti-obscenity forces.'*' At the same time, the 
National Advisory Council on the Arts, after meeting with NEA's General Counsel, unanimously 
declined to promulgate "decency standards" to aid in implementing the 1990 amendments.'"" To 
further assuage the doubts of the arts community, Frohnmeyer made several strong statements in 
speeches against government control over the content of publicly funded art."' Other observers were 
skeptical about the ability of the panels and the NEA Chairman to resist playing the censor's role. 
Professor Owen Fiss suggested that: "The risk remains great that, in the end. Helms will have his 
way, and grants will be denied by the endowment for projects like Mapplethorpe's on the ground of 
indecency, even though they cannot constitutionally be prosecuted criminally for obscenity."''' 
Several artists and arts organizations challenged the "decency" provision of the 1990 amendments, 
and in 1992 a federal district court in California found it to be unconstitutional.'" 

If Frohnmayer's conciliatory actions failed to persuade NEA's critics in the arts community, 
they infuriated its detractors at the other end of the spectrum."* When NEA awarded two more 
grants to members of the NEA Four, Congressman Dannemeyer publicly called for Frohnmayer to 
resign. "5 After Presidential Candidate Patrick Buchanan made NEA a campaign issue in the 1992 
Republican primaries,'" the White House quickly asked for and received Frohnmayer's resignation.'" 

The NEA controversy erupted anew when Frohnmayer's replacement, Candice Radice, declined 
to fund two applications that had been approved by the visual arts peer review panel and one that had 



unanimous peer-review approval was thai the Council and Frohmnayer feared that the work would be politically unacceptable.' [No 
author]. Politics Cost Gallery Grant, Director Says, Washington Post, Oct. 23, 1990, D7. 

Judith Reinraub, NEA Approves Delayed Grants, Washington Post, Jan. 5, 1991, CI. Frohnmayer personally reviewed the two 
applications to satisfy hinuelf that the granu met NEA's criteria. Id. Ms. Hughes was not especially grateful for the grant: 

I refuse to be surprised by anything the NEA does, li seems to me that the endowment is like a rudderless ship. This 
might get tossed my way, but I think other artisu whose identities are controversial— their race, their gender, their sexual 
orientation— are just going to be weeded out by the new internal changes in the NEA and the dissolution of the peer 
panels. 

Barbara Gamarekian, Arts Endowment Reverses a Stand, New York Times, Jan. 5, 1991, p. 9, col. 4. 

^lonan, supra note 368. 

"'See David Johnston, Lightning Bolts from Lcf\ and Right Can't Resist Arts Endowment Chief. New York Times. May 3. 1991. 
A12, Col. 1. 

Fisa, supra nou 298, at 2095-96. See also Owen M. Fiss, A Decency Czar: National Endowment for the Arts censorship. The 
Nation. Vol. 252. No. 14, p.473, April 15, 1991. 

'^FinJey v. National Endowment for the Arts. No. CV90-5236AWT. 1992 U.S. DIst. LEXIS 8070 (CD. Cal.. June 9. 1992); 
William H. Honan, 'Judge Overrules Decency Suiute for Arts Grants,* New York Times, June 10, 1992. Al. col. 2. But sec Advocates 
for the Arts v. Thompson. 532 F. 2 (d) 792 (Isi Cir.), cert, denied, 429 US 894 (1976) (upholding slates revocation of NEA-sponsored 
grant to journal after it published a controversial poem). 

^Leonard Garment, one of the co-chairmen of the Independent Commission, was critical of Frohnmayer's lack of political sensitivity: 

{A)fler the (1990] funding crisis passed, behavior ai the top of the Endowment did not change enough to make a 
difference. This fall Frohnmayer received a recommendation from the Endowmenl's Advisory Council that some of the 
avant-garde performance artists who had conspicuously angered endowment critics be given funds for more projects. He 
would not overrule it; the chairman's job— he gave me his principled explanaiion-was to overrule his council only if a 
reconunendation met a quasi-judicial sundard of clear and convincing error. The next day an Endownment suffer 
privately read me a string of phone messages received from staffers who were calling on behalf of the Endowment's most 
durable supporters in Congress. They all said more or less the same thing 'Has he lost his mind?' 

Leonard Garment, "The Feds and the Arts: Where it Went Wrong," Washington Post. February 25, 1992. AI7. 
"*Aian McConagha. Another Showdown Looms as NEA's Critiques Cry "Smut," Washington Times. November 6. 1991. Al 
'^^John W. Mashek. 'Buchanan Assails Sutra Nomination: Also Criticizes Arts Subsidies.* Boston Globe, February 21, 1992. p. 10. 
'^Editorial, Curtains for Mr. Frohnmayer, Washington Post, February 25, 1992, A 16. 
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been recommended by the peer review panel for sculpture.''* This prompted both panels to suspend 
their reviews of pending applications in protest.'^ Stephen Sondheim and Wallace Stegner turned 
down NEA medals of honor to protest the action, a grantee decided to turn his funds over to the 
rejected applicants,^ and it began to look like there would be a mutiny of peer review panels. 
Radice argued that her actions should not be viewed as an attack on the peer review system, and she 
insisted that they were not the result of White House pressure.'*' 

2. Feedback and Rebuttal 

Although an NEA fellowship or grant may not be as critical to the career of an applicant as an 
NIH or NSF grant, it is still a significant plum. An NEA fellowship can do a great deal to enhance a 
musician's or artist's status in the art world, and this can have long-range economic consequences for 
the artist. Consequently, rejected applicants are very interested to know why their applications were 
not funded. Although the agency's rejection letter is typically a form letter that contains very little 
substantive information, NEA encourages rejected applicants to contact the program specialists 
assigned to their projects for explanations and suggestions for the future. NEA staffer will usually 
provide reasons to rejected applicants who are considering an appeal, but only if specifically 
requested. A summary of the relevant panel's deliberations is available to any applicant that requests 
one,'*^ but staff notes of panel meetings and minutes of those meetings are not routinely shared with 
applicants or any other outsiders. Tapes of panel meetings are never shared with applicants or 
outsiders. Prior to 1990, the portions of the meetings of the Advisory Council that were devoted to 
discussions of individual applications were closed to the public, but the Council in May 1990 decided 
to open these sessions to public scrutiny.'" 

These efforts at providing feedback do not nearly approach the willingness of NIH and (more 
recently) NSF to provide reasons to rejected applicants and suggestions for improvement. There is, 
for example, no equivalent in NEA to the NIH "pink sheet" containing a summary of the panel 
deliberations that is routinely mailed to all applicants. NEA cites two primary reasons for its 
reluctance to provide reasons to rejected applicants as a routine matter: (1) the subjective nature of 
panel determinations in the arts; and (2) the extra workload that it would place on the staff.'*^ 
Although the latter explanation may have some merit, the former is unacceptable, and it goes a long 
way toward explaining recent criticism of NEA's grantmaking process. If the subject matter of a 
panel's deliberations is so subjective that the panel's reasons for funding one project and rejecting 
another cannot be articulated in a way that is understandable to the rejected applicants, the value of 
peer evaluation itself is open to serious question. There is a very fme line between an unexpiainable 
decision and an arbitrary one. Absent an explanation, neither the rejected applicant nor the public 
that pays for the funded projects can know whether the panel behaved arbitrarily or behaved 
reasonably in a way that could not be explained. In other words, NEA is demanding that rejected 
applicants and the public trust the process completely. In a time in which NEA comes under yearly 
attack from determined critics, this may be too much for the agency to expect. 



'^Maryann French, 'Second Arts Panel Walks Out," Washington Post, May 21, 1192. Dl; Blair Kamin. "In First Week, Acting 
Chairwoman Starts Steering NEA to the Right,* Chicago Tribune, May 10, 1992, p. 3. The rock group Aerosmilh came to the rescue 
with a $10,000 grant to fund the show. Patti Hartigan, 'Aerosmith Takes Aim al the NEA,* The Boston Globe, May 20, 1992. P. 1 . 

'Patricia C. Johnson, "NEA Sculpture Panel Suspends Work,* Houston Chronicle. May 16, 1992, p. 3. 

'"^Jon Robin Baiiz, a thiny-year-old playwright, accepted a SI 5, 000. grant from NEA and announced that he would make two 
donations to two institutions that were denied S10,000 grants. Alex Wiichel, *Ans Gram Winner Donates to Losers,' New York Times, 
June 2. 1992, Bl. Col. 1. See also, George F. Will, "Moral Preening as an Art Form." Washington Post, June 7, 1992, C7 (criticizing 
the action). 

'"Kim Masters, "NEA Chief Defends Grant Vetoes,* Washington Post, May 29. 1992. Dl. 

'^^omments of Michael McLaughlin, supra note 227. 

''^The Independent Conunission on the National Endowment for the Arts, supra note 232. 

'^^Commenu of Michael McLaughlin, supra note 227. 
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3. Favoritism 

As with all peer review programs, an NEA reviewer can play favorites by assigning a very high 
score to a particular proposal. In one NEA program, for example, a single panelist can guarantee 
that an application makes it to the third round of consideration by casting a "passion vote" for an 
artist for which he or she has a special fondness. Each panelist is informally given a limited number 
of such votes, and some use them more than others. Although these "passion votes" are intended to 
allow panelists to express especially well-formed opinions on the merits of the artist's work, they 
likewise offer an opportunity to enhance a particular artist's probability of success. Since an 
applicant must still gamer the support of five out of seven or more panelists in the final round of 
voting, however, there is a relatively low probability that a proposal lacking artistic merit will be 
recommended for funding. 

4. Old Boy Network 

Throughout much of its history, the NEA panel system has been criticized by outsiders for being 
an "old boy" network in which insiders impose their own arbitrary constraints on the art and artists 
who get funded.'" Interestingly, unlike much of the criticism of NIH and NSF, this criticism does 
not come so much from rejected applicants, as from outside observers who are generally critical of 
the substance of NEA funding decisions. These criticisms reached a fever pitch during the recent 
Mapplethorpe/Serrano and NEA Four controversies.^ 

One NEA staffer interviewed for this Report agreed with the contention of some outside critics 
that the agency is a "captive" of the Panels. If the Panels want to "look out for their own," there is 
little that the staff (or even the Chairman) can do about it. The high turnover rate of the Panels that 
is now required by the 1990 amendments, however, helps reduce the probability that a coterie of "old 
boys" will effectively control the Panel's decisions. Each panelist is chosen for only a one-year term 
and may not be appointed for more than three consecutive terms. Some panels have a 100 percent 
turnover rate. As one NEA staffer put it: "No matter who you are you are going to run out of your 
friends sooner or later." Another shield against "old boyism" is the agency's anempt to select panels 
that reflect geographic and cultural diversity.*^ Finally, the nature of the subject matter can help 



'^^See Allan Paracini, National Endowment: Arts Agency: Living Up To lis Billing? Los Angeles Tinies, Sept. 9, 1989, pt. 1, p. 1. 
col. 1 (citing Professor Kevin Mulcahy, Professor of Political Science at Louisiana State University for the proposition that the peer review 
system at NEA *has often tended to operate as a network serving the interests of well-connected artists that is hard for other anisu to 
penetrate.*); McLellan, supra note 8 (There are cliques and friendships in the Arts and questions of artistic merit often overlap with 
questions of personal taste.*); National Endowments for the Arts is Honored as 20th Birthday News, New York Times, Sept. 1, 1985, sec. 
1 , pt. 2, p. 68, col. 1 (The endowment has an insider-outsider working system. In a time of no expansion in the arts, younger artists are 
coming up against a nniddle-aged generation of arts administrators who make up the grant-giving panels.*) (quoting Mr. Derek Guthrie, 
publisher of The New Art Examiner); Can the Government Promote Creativity— Or Only Artisu? New York Times, April 25, 1982, bcc. 4, 
pg. 6, col. 1 (*Decisions . . . about what's good and worthy of support are made centrally by people who sit on peer review panels who do 
represent one segment of the society. They are almost 99% previous or current grantees.*) (quoting Ms. Arlene Goldbard, Co-director of 
the Baltimore-based Neighborhood Arts Programs National Organizing Committee); Lanquette, supra note 226 (*[T]he charge most often 
heard today is that these panels, rather than acting as government arbiters can work instead as 'old-boy' networks— passing out plums to 
friends and colleagues.*); Richard Netzer, The Subsidized Muse (Cambridge University Press, 1978) ('Consciously or not, some (peer 
review] panels amount to 'old boy' networks that respond favorably to applicants who are part of that network.*). 

''^See, e.g., Robert H. Knight and John M. Slye, The National Endowment for the Arts: Misusing Taxpayer's Money, Heritage 
Foundation Reports, Jan. 18, 1991, Backgrounder number 3 (*The peer-review process is a tightly-knit buddy system, with artists taking 
turns giving each other grants'); Jonathan Yardlcy, Helms and the Art of Pragmatism. Washington Post, July 31, 1989, C2 (*Rather than 
represent the public's interest, too many of these panels represent those of their own artistic and scholarly cliques, they dole out money to 
allies and proteges, feather their own nests and keep it all in the family.*). 

Attempts to reflect geographic diversity stem from a rather consistent and strong perception among educated outsiders that the 
National Institutes for the Arts and Humanities are biased in favor of East Coast cities and schools. See, e.g., Paracini, supra note 385 
(citing several prominent supporters of NEA who nevertheless believed that the Northeast has dominated in the competitions and for the 
proposition that NEA has not achieved broad ethnic representation). One especially strong adherent to this view was nominated, but not 
confirmed to be the Chairman of the National Endowment for the Humanities. He vowed that if confirmed, he would ensure that *|n]ot 
everything would go to Harvard, Yale, Princeton, and Chicago* and that 'more grants (would go) to Texas and Oklahonu* Carla Hall, 
Bradford Speaks Out: NEH Chair Candidate on His Program Plans; Bradford on the NEH. Washington Post, October 28, 1981, Bl. 
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prevent old boy networks from arising. Unlike scientists, artists and musicians are not so narrowly 
specialized that only twenty or thirty people in the country are qualified to judge the merits of all the 
proposals in a given field. For example, the available pool of experts to judge a saxophone ensemble 
probably includes saxophone professors from most of the major and many minor universities, as well 
as most professional saxophonists. 

Despite the changes, critics maintain that NEA continues to administer an old boy network that 
sends public monies to avant guard projects.'** The nub of their criticism is that the NEA staff is 
responsible for selecting peer reviewers who will predictably reach predetermined results. In other 
words, the critics accuse the NEA staff of "stacking" the peer review committees with advocates of a 
single point of view. ^^ While it is difficult to evaluate the validity of this claim, there are few built- 
in constraints against such manipulation of the peer review process. 

5. Halo Effect 

The "halo effect" identified at some of the scientific agencies seems to be less pronounced at 
NEA. In part, this is attributable to the very large number of applicants and the absence of a 
tradition of continuously funding any single individual applicant's grants. Only a very small number 
of individual grantees are funded for a second consecutive year. It also may stem from the fact that 
well-known artists are often much more highly paid in the private market and therefore do not stake a 
claim to NEA funds or do not vigorously devote themselves to the application process. According to 
one NEA staffer: "I have witnessed extremely well-known artists' work rejected because they had 
sent in slides of old work." Although panels are not supposed to consider the applicant's financial 
need in evaluating the quality of proposals, there is an unwritten rule that artists who do not really 
need the money should not ask for NEA's limited fellowship funds. The halo effect may, however, 
play a larger role in programs involved in funding large organizations, such as orchestras and 
museums. 

6. Personal Animus 

NEA has received frequent co.nplaints that the Chairman or the Council has rejected applications 
because of a dislike for the applicant or some aspect of the applicant's lifestyle. During the NEA 
Four controversy, for example, applicants alleged that several actions taken by the Council and the 
chairman were motivated by animus against their homosexual life styles. Similar complaints have 
not, however, been lodged against the peer review panels. 

7. Mavericks 

NEA does not appear to have a problem with refusal to fund mavericks. Highly original 
projects are routinely funded, and creativity is perceived as a great virtue in the panels. The agency 
suffers considerably more criticism from the avant garde nature of some of its funding decisions. 
Indeed, some panel actions, such as the recommendation to fund a former prostitute named "Scarlet 
0" who decided that she would rather be known as a performance artist,'* call into serious question 
the very professionalism upon which the peer review process critically depends.'" 



'^'See text accompanying notes 305-307, supra. 

'"Frcquenl NEA critic, Richard Grenier claims that "Peer review panels' . . . an sucked, a joke.' Richard Grenier. A Vote to 
Bring Back Guillotine at NEA, Washington Times, April 4, 1991, pi. G, p. 61. Grenier ofTers no empirical support for this proposition, 
but it seems to be widely held by culturally conservative critics of NEA. 

•***Todd Allan Yasui, I>efending NEA's Velos. Washington Post, February 10, 1992, 87 (in a videoupe of one of her performances, 
Ms. O discussed gender, stripped, and invited members of the audience to rub lotion on her body). 

'"See Walter Bems, Saving the NEA, National Review, Vol. 42, no. 22, p. 34, Nov. 19. 1990 (complaining that "the Arts 
community insists that everything produced by someone claiming to be an artist is in fact a work of an.*). 
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8. Applicant Anonymity 

Some programs at NEA attempt to maintain applicant anonymity to some degree. In some fields 
that have a tradition of "blind judging" and in which applicant anonymity is practicable, that 
technique is often used to provide an additional element of objectivity to the grants process. For 
example, peer review panels for some music program typically rely upon tapes of applicants' 
performances in which the identities of the performers are not revealed until after all of the tapes 
have been evaluated and compared. According to one NEA staffer, the pendulum appears to be 
swinging toward more blind judging in all of the programs where it is feasible. 

In other areas, the panelists resist blind judging. For example, applications in the area of 
paintings could probably be evaluated blindly. The quality of submitted examples of the artists' 
works could be evaluated without necessarily knowing the name of the author. The panelists, 
however, believe that since most artists who apply for grants are not known by name to the panelists, 
the evaluation is essentially anonymous in any event. In addition, blind evaluation could work 
mischievous results in cases of highly derivative works in which the student's works may be mistaken 
for the teacher's. Finally, blind evaluations do not allow Panels to take into account subtle timing 
questions involving the impact that support might have on the artist's work at some particular stage in 
the applicant's career. 

9. Reviewer Anonymity 

The names of each year's group of around 800 panelists are made public at the time the panelists 
are appointed, but NEA does not otherwise routinely publicize the identities of the Panel reviewers. 
The agency will, however, make the identities of panelists available upon request after the reviews 
are completed. Panels that use "primary reviewers" to conduct intense reviews and report back to the 
panel, however, do not reveal the identities of primary reviewers to applicants. All panel meetings 
and, until recently, portions of meetings of the National Council on the Arts were held in private. 
This secrecy has always engendered criticism from rejected applicants.'^ But is vigorously defended 
by the professionals that sit on the panels.'" 

10. Financial Conflict of Interest 

Prior to the 1990 amendments, NEA received a great deal of criticism about the extent to which 
panelists encountered at least indirect conflicts of interest.'*' NEA's regulations prohibited an 
applicant from sitting on a panel that considered the applicant's proposal, but they did not prevent the 
a panelist from sitting on a panel that considered an application from his or her institution, so long as 
the panelist left the room during the consideration of that application."* NEA revised its conflict of 
interest regulations in light of the 1990 amendments to prevent any person who is an applicant or a 
representative of an organizational applicant from serving on a panel that considers that application. 
In addition, persons who are employed or associated with an organizational applicant may not serve 
on the panel that evaluates its application. Panelists may receive remuneration from activities 
supported by NEA only to the extent specified in the application prior to its consideration, and they 
may not participate in decisions with respect to that application. 



''^lueck, supra note 294 ("Endowment applicanu have long complained aboul the secrecy of the panels.*). 

'"See Beeb Salzer, Politicians Need Maturity, Tolerance, and Strength in Nurturing An, Los Angeles Times, July 24, 1988, pt. 2, p. 
2, col. 1 (Ms. Salzer is a Professor of Drama at San Diego State University) ('All of the ranting, pleading, negotiating, and evaluations 
that are part of the selection for art's grants are best handled by experts sway from the spotlight.') 

"^See Knight and Slye, supra note 386; Joyce Price, NEA Grant Procedures Need Fixing, Says Panel, Washington Times, Sept. 11, 
1990, A5; Paracini, supra note 306. Similar conflicts have been noted in sute arts councils. See Chris Pasles and Herman Wong, Arts 
Council Doling Out Funds Today, Los Angeles Times, Aug. 28, 1987, pi. 6, p. 1, col. 2. 

"^NEA Sundards of Conduct, supra note 277, at 6. 
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11. Research Conflict of Interest 

Prior to the 1990 Amendments to its statute, NEA's regulations did not prevent a person 
employed by an applicant from judging the application of competing organizations. Even though not 
present when the application from his or her organization was being considered, the panelist still 
knew that a low rating for the competitor would leave the competing organizations in a relatively 
better position overall. According to one high level NEA staffer, NEA received applications from 
"dire competitors," and although the panelists attempted to police themselves, the appearance of 
impropriety was hard to avoid when the organization whose employee was on the panel received a 
large grant, and the competitor was not funded. 

The agency staff attempted to deal with such conflicts informally. For example, on one occasion 
when the Assistant Program Director observed a panelist intensely criticizing a proposal from a 
competitor organization, the Assistant Director attempted to steer the discussion away from the 
criticisms. When this failed, the Assistant Director suggested an early lunch. After lunch, the 
Program Director joined the panel and said nice things about the proposal to offset the previous 
negative comments, and the proposal was recommended for approval. 

The 1990 Amendments met the problem of research conflict of interest head-on by flatly 
prohibiting a person from sitting on a panel that evaluates the applicant's proposals or those from the 
institution that employs the applicant. Although this necessarily reduces the supply of qualified 
experts for panels, Congress correctly concluded that it was a necessary step to preserve the public 
perception that the programs were being run fairly. The informal approach depended upon the ability 
of the panelist to detect the threat to impartiality posed by his or her consideration of a proposal from 
an organizational competitor and upon the applicant's willingness to do something about it. The staff 
attempted to repair in an ad hoc and informal way the damage to the process caused by panelist 
improprieties, but this depended too heavily upon the ability of the staff to identify the conflicts, and 
it was not always be feasible to offset the effects of the prior bias. 

The other aspect of research conflict of interest, (appropriation of ideas) may not be as fully 
applicable to the arts as to the sciences. It is, of course, possible for an artist or musician to steal an 
idea from another and use it to his or her own advantage.'** But it would be difficult to arrange such 
a theft in the context of the NEA grant application process, because the projects are either simply 
fellowships to pursue a broad activity or discrete projects that are described with such generality that 
the kernel of the underlying Idea usually cannot effectively be appropriated. 

12. Peremptory Challenges 

NEA does not currently let applicants review lists of potential panelists to identify persons with 
a potential bias. One very high level NEA official interviewed for this Report, however, expressed 
the belief that "peremptory challenges" could go a long way toward ensuring against bias in the NEA 
peer review system. The concept could be relatively easily implemented in most NEA panels, 
because they are chosen from a large number of prospective panelists. With a large number from 
which to choose, the quality of the resulting panels should not suffer significantly from the 
implementation of a process allowing peremptory challenges. 

13. Lobbying 

Most institutional applicants place many of the Council members on their mailing lists to receive 
promotional literature, newsletters, etc., but they do not often make direct overtures to Council 
members with respect to pending grant applications. Rejected applicants occasionally communicate 



^^When asked why prominent scholars would voluntarily devote so much time to serving on NEH panels, one scholar volunteered: *I 
find this an excellent way of finding out what's going on in research. Even if a proposal isn't good, I learn something about that field 
from the discussions with the other panelists. And serving is part of my professional responsibility to other scholars.* Felicity Barringer, 
Scholars Weigh the Allocation of Billions, Washington Post, September 11. 1981, A17. Although the quoted panelist did not suggest that 
he would attempt to steal another's ideas, it does suggest that the potential is there, even in less commercial world of the arts 
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with panelists after-the-fact, though rarely before the application is considered. The staff prefers that 
disappointed applicants deal directly with the staff, and not "go over their heads" to Council 
members. Applicants respect this wish for the most part. 

14. Political Pressure 

NEA receives many letters from congresspersons and others in support of particular projects, 
but the staff does not immediately share these with the panelists or Council members. They are 
usually placed in the applicant's file and answered with a polite reply describing NEA's peer review 
process. Because the challenge grants are much larger, outsiders attempt to communicate with the 
Challenge Grants Program Director more than with other Program Directors. They do not, however, 
use arm twisting tactics. Any letters received from outsiders, including congresspersons, and 
memoranda of telephone conversations are placed in the applicant's file, and they may be read by 
panelists and members of the Advisory Council. In practice, they rarely are consulted in the vast 
majority of NEA programs. On the other hand, the NEA Four controversy clearly demonstrated that 
political considerations do play an important role in decisions about controversial applications that 
may be considered indecent or offensive. 

15. Efficacy of Appeal Procedures 

Rejected applicants may formally appeal to a Deputy Chairman of NEA, but only on relatively 
narrow grounds.'^ Appeals based on artistic merit are not entertained. In practice, most appeals are 
of the latter variety, and are therefore given short shrift. Tapes of pane! meetings are preserved for 
two years for use in possible appeals. From the tapes, the staff can glean whether the criteria that 
guided the Panel were within those detailed criteria that the agency published in advance. ** It is not 
clear, however, how the applicant can directly ascertain whether irrelevant criteria were used. 

In cases were the applicant does discover that irrelevant criteria were used, the Chairman can 
reverse the Panel. For example, an applicant for a music performance grant for a saxophone 
ensemble obtained a summary statement of the panel's deliberations and ascertained that a primary 
reason for rejection was the panel's belief that the ensemble should only play music that was 
originally written for the saxophone, and not transcribed from music written for other instruments. 
Since this was not a relevant consideration under the Program's published criteria, the Deputy 
Director sent the application back to the panel for reconsideration at its next meeting. One irrelevant 
criteria that occur with some frequency is whether the applicant organization has a "need" for the 
money. 

Apparently, there is an additional informal appeals procedure for decisions by the Chairman not 
to fund a project that has received the endorsement of both the relevant peer review panel and the 
National Council on the Arts. In the first case in which the Chairman took this action, vetoing a 
proposal to use plants that absorb toxic metals to clean a hazardous waste site,*** Chairman 
Frohnmayer agreed to meet with the rejected artist to discuss the reasons for his action. After the 
meeting, Frohnmeyer reversed his decision, explaining that he had been persuaded that the project, 
which initially seemed more appropriate for EPA funding, had artistic value.*" 

16. Conclusions 

It seems reasonably clear that the controversy over NEA is not over. The tension between the 
inevitably elitist concept of artistic excellence and constant demands for democratic accountability is 



'^See text accompanying notes 289-292, supra (description of formal appeals process). 

'^Probably the most frequently relied upon irrelevant criterion is need. It is not for the panels to decide whether the applicant really 
needs the money or is already wealthy enough. 

'"William Honan, U.S. Arts Chief Overturns an Approval, New York Times. Nov. 27, 1990. B3, col. 1 . 

^''^m Masters, NEA Grant Reversal Scene: Chairman Set to Approve Project He Rejected, Washington Post. Dec. 21, 1990, D2. 
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likely to continue as long as the agency remains willing to fund grants in controversial areas. 'Hie 
peer review model is at the heart of the decisionmaking process and will therefore no doubt remain at 
the center of the controversy. More than any other agency studied for this Report, NEA's peer 
review procedures have been criticized in the press and in other public forums such as congressional 
hearings. Yet past and present NEA officials continue to defend the peer review process that has 
evolved within that agency. Former NEA Chairman Frank Hodsell noted: "We may not have a 
perfect balance, but we have several checks. Program directors submit two, three, or four names for 
every slot; we look at them from the point of view of geographical distribution, we try to ask all the 
right questions. But we never do a perfect job."*" One high level staffer argued that although the 
agency has an elitist image in the popular press, it in fact serves a very broad public primarily at the 
local level: "In fact, our grants touch everything and everyone. A grant to an artistic program of any 
type in a community spreads and affects the whole community, becomes part of the educational 
system." 



*'Glucck, supra noie 294. 
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VI. THE POTENTIAL FOR BIAS IN PEER REVIEW 

Although the funding decisions of the scientific and artistic granting agencies in the United 
States are theoretically bound by objective criteria, the preceding chapters have demonstrated that 
there is room for bias in the process. The extent to which bias actually affects individual outcomes is 
a difficult empirical question that has been examined periodically, but has never been resolved. 
While this Part does not purport to answer that question definitively, it will draw upon the existing 
literature and th^ experiences of the four agencies related in Parts 2-5 to explore the potential for bias 
in the peer review process and to suggest the ways in which bias in the system can yield undesirable 
outcomes from a societal perspective. 

Nearly all of the researchers and grants administrators interviewed in connection with this 
Report agreed that bias was not a pervasive problem in NIH, NSF, and EPA.*" The evidence of bias 
in NEA is somewhat stronger. For a long period of time that agency allowed panelists to sit on 
panels that reviewed applications from their own institutions and from their friends and close 
associates, and some outside critics discerned a strong pattern of "old boy ism" in the outcome of the 
granting process.*" Yet most complaints of bias in the peer review system at NEA have come more 
fi"om outside critics who are dissatisfied with the results and may have their own axes to grind. 

The lack of direct empirical evidence of actual bias in the granting agencies studied here is not 
especially surprising, because bias is not especially susceptible to empirical validation. Even the 
potential for bias, however, bears careful examination, because the public perception that a grants 
process in which public funds are at stake is biased can erode public support for the program and 
ultimately result in its decline or demise. The following discussion reviews the categories of bias 
that were developed in Parts 2-5 and suggest how bias can detract from sound decisionmaking and 
reduce public support for federal grant programs. 

A. Animus 

In a grants program that is able to fund only a small number of meritorious applications, a single 
bad review can prove fatal to a grant proposal.*** The person who is the object of personal animus on 
the part of a peer reviewer is obviously adversely affected, because his or her chances of receiving 
the discretionary grant are thereby reduced or eliminated. That person can legitimately complain that 
he or she has been treated unfairly. In a broader sense, animus runs counter to the public interest in 



After an intensive investigalion into the peer review process at NSF in 1976, a House investigatory subcommittee concluded: 

No method superior to peer review has been found for judging the scientific competence of proposers. Scientific peers are better able 
than others to judge the design of proposed work, the importance of proposed work to the scientific field, and the past performance of the 
proposer. Appropriate peer review procedures generally lead to the support of proposals in a high quality range. Using peer review 
procedures the Foundation has successfully fostered significant advances in basic science over the past 25 years. 

1976 House Hearings Report supra note 3. A more recent report of a 'Merit Review Task Force* assembled by NSF concluded: The 
system is remarkably fair and is an effective mechanism for identifying and funding high-quality proposals in every discipline the 
Foundation supports. Many consider it to be the best system in the world.* Report of the Merit Review Task Force supra note 100, at 1 . 

Two perceptive observers of the peer review process in the scientific agencies, Chubin and Hackett, accurately complain that the 
criticisms of peer review that periodically surface in the popular media are both frustrating and unsatisfying: 

They are frustrating because they reflect limited view points or special interests that do not provide a comprehensive perfective. 
They are dissatisfying because, in their brevity, they address issues selectively and often offer scant empirical or logical support. Even the 
best of them are but a lone scientist explaining how he or she has been mistreated, then generalizing from thai experience to a set of 
universal principles. 

Chubin & Hackeu, supra note 2, at 33-34. 

^^ScePartS.aln. 162. 

***A8 one very successful NSF applicant observed, *(a]ll one needs is one unsubstantiated or emotional comment by one reviewer to 
significantly lower one's chances.* Chubin & Hackett, supra note 1, at 78. Another close observer of the peer review process noted that 
*in the present climate of opinion, a colleague who knows that he or she has the certain power to doom [b| proposal by a checkmark in the 
"Fair* or *Good" category - even if accompanied, albeit inconsistently, by written praise - might well be inclined to use it." R. Roy, 
Alternatives to Review by Peers: A Contribution to the Theory of Scientific Choice, 22 Minerva 316, 319 (1984). Many other applicants 
and reviewers interviewed for this report expressed similar views. 
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awarding grants to the most deserving applicants through an objective application of the statutory or 
regulatory criteria. A single, isolated case of animus will probably not prove very damaging to 
programs in which many qualified applicants contend for a limited supply of discretionary funds. But 
if the bias is manifested against whole groups of people or if a significant number of those in the 
potential pool of applicants perceive that the system could be unfairly abused to discriminate 
irrationally against them, potential applicants may be discouraged from applying for grants in the 
future. More subtly, in order to avoid a single negative assessment, proposal writers may begin to 
submit bland proposals for inoffensive studies that do not advance scientific knowledge or artistic 
creativity to any significant degree.*" Similarly, public support for a system that consistently 
blackballs qualified applicants for reasons unrelated to the statutory or regulatory criteria will 
probably erode over time. Fortunately, the description of the peer review process in Parts 2-5 
revealed very little evidence of personal animus in the agencies studied here.*^ 

There is an animus of a less personal sort that seems to pervade the peer review process in the 
scientific agencies. Peer review panels in NIH and NSF are often accused of being biased against 
"maverick" proposals that reject the conventional wisdom of the mainstream scientific community.*" 
Thomas Kuhn has observed that scientific revolutions occur when a few scientists begin to reject the 
dominant "paradigm" or conventional wisdom of the scientists who work in the field. The 
conventional paradigm is accepted in the literature and in the classroom, and it produces the 
background assumptions that facilitate scientific dialogue. As bits and pieces of data are reported that 
cannot be explained by the conventional paradigm, however, a few practitioners are forced to re- 
examine the background assumptions and some begin to reject the paradigm, at least in some 
contexts. Adherents to the old world view i: variably resist challenges to the conventional paradigm, 
but sooner or later the mounting evidence produces a scientific revolution in which the dominant 
paradigm is rejected and a new paradigm takes its place.** 

An important battleground in the war between the paradigms is the discretionary grants process. 
The most frequently identified bias among the scientists interviewed for this report was the subtle 
bias that the scientific peer review bodies exert against "mavericks" who reject assumptions and 
approaches of the dominant paradigm.*^ Nearly all observers of the process agreed that mavericks 
have a much lower probability of receiving ftinding than those who submit proposals well within the 
bounds of conventional wisdom. This is not so much a matter of personal animus against particular 
applicants as it is a matter of ideological self-defense. People who have spent their careers 



^'^Chubin & Hackett, supra note 3, ai 76 (The lesson is clear, ... a wriler must please all of the reviewers all of the linte, and be 
especially careful never to ofTend any of them.*). 

^Congressional hearings into the peer review process at NSF conducted in the summer of 1976 produced some limited testimony of 
the existence of 'bias against someone a reviewer knew and disliked or disagreed with.* 1976 House Hearings Report, supra note 3, at 
33. NEA has received some complainu that the NEA Chairman or the Advisory Council on the Arts has rejected applications because of a 
dislike for the applicant or some aspect of the applicant's lifestyle, but these accusations are virtually never directed at the peer review 
panels. 

*"ln a 1986 survey conducted by NSF of nearly 10.000 acadenuc scientists, almost two-thirds agreed with the suiemeni that *NSF is 
not likely to fund high-risk exploratory research because the likelihood of obtaining favorable reviews is slim.* National Science 
Foundation, Final Report: NSF Advisory Committee on Merit Review (1986). See also BJerklie, supra note 105. In a survey successful 
and unsuccessful applicants at the National Cancer Institute, 60.8% agreed with the proposition that reviewers were relucunt to support 
unorthodox or high-risk research, while 17.7% disagreed and 21.4% were neutral. Chubin & Hackett, supra note 1, at 66, Table 3. A 
former Vice-President for Research at the Massachusetts Institute of Technology ustified in 1979 that the peer review process discriminates 
against new interdisciplinary science and scientific thinking that is not *au courant' even though creative and ripe with 'unusual 
possibilities for breakthroughs.* L. Carter, A New and Searching Lxxik at NSF, 204 Science 1064 (1979) (quoting testimony of Dr. 
Thomas F. Jones, Vice President for Research at MIT). See also 1976 House Hearings Repon, supra note 3, at 27 (*If a proposal 
challenges the mainstream of scientific thought, the expert peer reviewer who is in the nuinstream will tend to see the proposal as wrong 
on the face of it.'). 

**Kuhn, supra note 13. 

^''By contrast, persons interviewed in connection with this Report and outside observers of NEA reported no dissatisfaction 
whatsoever with NEA's treatment of mavericks. Indeed, many complaints went in precisely the opposite direction— that the agency funded 
too many 'off-the-wall" proposals with little chance of success. 
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conducting research aimed at bolstering and extending the dominant paradigm are reluctant to direct 
resources toward research aimed at destroying that paradigm. 

B. Tunnel Vision 

Just as peer review panels tend to be biased against mavericks, they often exhibit tunnel vision 
with respect to whole categories of proposals that appear to lack relevance to professionals trained in 
a particular discipline or subdiscipline. This "tunnel vision" problem is partially a function of the 
composition of the peer review committees and partially a problem of ambiguity as to any particular 
panel's proper substantive bailiwick. The discussion in Parts 2-4 indicated that tunnel vision does 
appear to exist in the scientific agencies studied for this report. 

C. Favoritism 

Favoritism adversely affects the public interest in several important ways. First, if favoritism 
plays a significant role in overall proposal ranking, the awards will not go to the most deserving 
applicants, thereby undermining the policies underlying the grant program. Second, if the perception 
grows that a grant program plays favorites, ftiture applicants may be discouraged from applying. 
Finally, as the perception of favoritism spreads from the participants in the process to the public at 
large, public support for the program will no doubt dwindle. 

1. Old Boyism and the Halo Effect 

One of the most common complaints raised by participants in the peer review grants system is 
that it fosters an "old boy network" that subtly conspires to dominate the agency's evaluations. The 
closely related "halo effect" exists when peer reviewers fund poorly conceived projects by well- 
known scientists or scientists from highly regarded institutions purely out of regard for their past 
reputations, and not on the merits of their current proposals. Although complaints about "old boy" 
networks and the halo effect are frequently heard,*'" the empirical basis for such claims is relatively 
weak.*" 



''^'a Bununary of the testimony presented at extensive 1976 congressional hearings on the granting process at NSF concluded that: 
The most common objection to the use of peer review is that it is subject to 'baclcrubbing' or 'old boys clubs,' in which mutual friends 
unduly praise each other's proposals. ° 1976 House Hearings Report, supra note 3. at 22. In a 1986 survey of about four thousand 
scientists conducted by a national science fraternity, 63 percent agreed with the statement: 'Procurement procedure for grants to do 
govemmentally sponsored research depends on 'who you know.' Many requests seem to be funded primarily because the researchers are 
already known to and supported by the granting oi;gamzations.* Sigma Xi, A New Agenda for Science (preliminary report) (1986), 
reported in United States General Accounting Office, Univereity Funding: Information on the Role of Peer Review at NSF and NIH 7 
(March, 1987). In a survey of successful and unsuccessful applicants for National Cancer Institute granu, 39.5% of the respondenu 
agreed with the proposition that 'old boys networks' controlled the study sections, while only 32.7% disagreed and 27.8% were neutral. 
Chubin & Hackett, at 66, Table 3. The most strident critics of the peer review system characterize it as 'an incestuous 'buddy system' that 
frequently stifles new ideas and scientific breakthroughs, while carving up the mullimillion dollar Federal research and education pie in a 
monopoly game of grantsmanship.' 1976 House Hearings Report, supra note 3 (testimony of Rep. John Conlan). Similar complaints are 
directed at funding in NEA, but mostly from critics who are not participants in the peer review system. 

^"A 1988 NSF study of 95(K) principle investigators found some slightly suggestive evidence of a perception of favoritism anK>ng 
NSF grantees. Consistently successful applicants were 'more likely than the average applicant to be male, older .... and much more 
likely to be associated with a [prestigous research] institution,' and 97% of them had served as reviewers or panelists, an indication of high 
status in the relevant research community. National Science Foundation, Proposal Review at NSF: Perceptions of Principal Investigatora 
10 (1988) fhereinafter cited as NSF Perceptions Report]. See also J. McCuUough, First Comprehensive Survey of NSF Applicanu 
Focuses on Their Concerns About Proposal Review, 14 Science, Technology, & Human Values 78, 81 (1989). At the same time, 
consistently successful awardees 'were more likely to have known the program officer and to have itiade some personal contact before 
submitting a proposal.* McCuliough, supra, at 81; NSF Perceptions Report, supra , at 12. When the respondents who indicated 
dissatisfaction with the process (38% of all respondents) were asked to give reasons, they were somewhat more inclined to attribute any 
failure in the process to incompetent reviewers (18%) than to any cronyism, politics or an old boy's network (12%). NSF Perceptions 
Report, supra, at 14-16; McCuliough, supra, at 82. In an independent survey of 719 applicants for NIH grants in fiscal years 1980-1981, 
17.9% strongly agreed and 23.5% agreed with the proposition that 'old boy networks control the RIGs or study sections,' while only 
23.1% disagreed and 9.7% strongly disagreed. G. Gillespie, Jr., D. Chubin & G. Kurzon. Researchers' Cynicism and Desire For 
Change, 10 Science, Technology & Hunun Values 44, 45 (Summer 1985). 
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A granting system run by a coterie of old boys discourages innovative proposals from bright 
applicants who have not found their way into the network. In the extreme, science and art are 
reduced to schmoozing. Old boy networks do not give all applicants a fair opportunity to secure 
funding, and a system that is not fairly run will gradually lose public support. It is less clear that the 
halo effect is counterproductive or unfair. To the extent that her halo has been earned by the 
production of excellent work over many years, the beneficiary of the halo effect is being given an 
opportunity to continue her productivity despite the fact that her most recent proposal is not well 
written or is poorly conceived. Some critics of the peer review system complain that it rewards the 
ability to write good proposals, rather than the ability to produce good research or art.*'^ These 
critics believe that limited governmental funds should be dispersed in large multi-year grants to a 
very few scientists or artists who have the very best reputations in the relevant fields. They would, 
in other words, elevate the halo effect to a matter of principle. Others argue that an applicant's 
reputation should play a role in the process, but the dominant considerations should still be quality of 
his or her most recent efforts and the content of the current proposal.*'' To do otherwise is to give 
hallowed elders an unfair advantage. In sum, whether agencies should take steps to reduce or 
eliminate the halo effect is a fairly debatable point. 

2. Stacking the Deck 

The agency staff can play favorites by "stacking the deck" with reviewers who will predictably 
fiind the staffs favorites.*'* The extent to which staff discretion in choosing panelists is a good or 
bad thing is, like the halo effect, a matter about which reasonable minds can differ. Most would 
agree that a panel that is a stacked deck will not necessarily make objective funding decisions and 
will be perceived as unfair. The staff has an obligation to ensure that panels are balanced, and indeed 
this may even rise to the level of a legal duty under the Federal Advisory Committee Act.*'^ But one 
person's attempt to achieve balance may be another person's plot to stack the deck. It is, in other 
words, very difficult to know whether particular appointments are aimed at ensuring funding for 
favorites or for remedying imbalance in a committee already wedded to a particular point of view. 

D. Conflict of Interest 

1. Financial Conflict of interest 

The scientific and arts communities apparently recognize the judicial principle that no person 
should be the judge of his or her own case, and regulations in all of the agencies studied in this 



The empirical basis for the halo efTect is also weak. A five-year study of NSF grants conducted during the late 19708 found no 
evidence that reviewers at 'major* research institutions treated proposals from applicants at major institutions more favorably than did 
reviewers from lesser institutions. In fact, there was *a tendency in the opposite direction.* Cole, Cole and Simon, supra note 126. The 
■ame study found only 'low or moderate* correlations between actual funding at NSF and other indicators of the halo effect including 
academic rank, locus of Ph.D training, and previous NSF funding history. 

*'^ce Roy, supra note 6, at 73-81; Roy, supra note 404. 

*''Thi8 was the position taken by most of the agency staffers questioned on this issue. 

*'*Several witnesses at the 1976 House subcommittee hearings on peer review in NSF agreed that agency suffers could achieve rough 
ranking outcomes by choosing particular reviewers with known predispositions, but there was little agreement on the extent to which this 
was in fact happening at NSF. 1976 House Hearings Report, supra note 3, at 31 . According to one critic of peer review: *(t]he sysum is 
susceptible to manipulation by managers who operate under the emotional cover of the peer review' rubric so that individual proposals 
may be favored or disfavored simply by appropriate selection of the reviewers.* Roy, supra note 6, at 73-81. 

In agencies like EPA in which the chairman of the peer review committee chooses the panel members, the staff do not have an 
opportunity to stack the deck. The person that chooses the panel members, however, may still suck the deck with persons who are likely 
to fund his or her favorites. This may be more disturbing from a public policy perspective, because the entity exercising the most 
discretion is not an accounuble government employee. Indeed, at the extreme, such delegation of govemmenul power to private 
institutions may raise constitutional concerns under the delegation doctrine. See Carter v. Carter Coal Co., 298 U.S. 238 (1936). 

*'*See Part 7, infra (discussion of applicability of Federal Advisory Committee Act to peer review panels). 
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report preclude applicants from participating on panels in which their own proposals are evaluated.*'* 
Like animus and favoritism, financial conflict of interest can damage the public interest by 
discouraging qualified applicants from entering what appears to be a "rigged" system. A public 
perception that researchers are feathering their own nests with federal research dollars would almost 
certainly lead to reduced public support of government-funded research. Finally, it is simply 
immoral (and perhaps illegal) for an individual to appropriate another's ideas through the peer review 
process for private gain. Theft is theft, whether it takes place in the streets or in a conmiittee room. 

Most granting agencies are very sensitive to the financial conflict of interest problem, and they 
have promulgated regulations to shield against it.*'^ The regulations typically ensure that no peer 
reviewer is in a position to review his or her own proposal or. a proposal in which he or she has a 
financial stake. The regulations, however, do not always prohibit reviewers from evaluating 
proposals that are in direct competition with their own present or future applications, a matter that 
will be discussed in the next section. 

2. Research Conflict of Interest 

Even if the potential grantee recuses himself from the decision on his particular grant, he may 
have a financial stake in his evaluations of the remaining competing applications. Under the scoring 
systems that are typically used for allocating limited funds among members of a pool of qualified 
applicants, the chances of the reviewer's proposal being funded are enhanced if the scores of the 
competing proposals are held down."'* In highly competitive fields like chemistry and biotechnology, 
any additional advantage in the great race to make a new discovery is useful."" The peer review 
process gives researchers access to the innovative ideas of other scientists and artists in their fields, 
and they can use those ideas in their own research. Indeed, it has been suggested that unshared 
access to the contents of grant proposals is one of the few small bits of compensation that a reviewer 
receives for the many hours that he or she devotes to the review process.*^ 

The argument for allowing scientists to review competing proposals is that in some very narrow 
fields, there are literally no other people in the world qualified to provide an adequate peer review."^' 
Under the well-recognized judicial doctrine of "necessity," a judge may hear a case in which she has 



"'*N1[H Manual 4510, supra note 51; 45 C.F.R. §684. 11(g); NSF Proposal and Award Manual 1-5 (1989). The proposals of panelisu 
in NIH are reviewed by ad hoc panels assembled for that purpose. Id. at 5. 

*"Sce Part 2, at note 63; Part 3, at note 33. Part 4, at note 33; Part 5, at note 53. 

"'^See NSF Perceptions Report, supra note 411, at 24 (*ln times such as the present, where the success rate on prc^osals becomes 
ridiculously low, the reviewers all become nervous and supercritical because they know that a positive review severely diminishes their 
own chances of funding.*); Roy, supra note 6, at 75 ('Because of the absence of even simple procedures to avoid conflict of interest, 
scientists are often asked to judge the work of competitors, knowing that an excellent rating might jeopardize their own funding.*) 

Critics of the NEA granu process also complained bitterly of the fact that panelists could sit on panels that judged applications from 
Iheir own institutions so long as they absented themselves from the consideration of those applications. These complaints inspired 
Congress to provide in the agency's statute that no person could sit on a panel that considered an application from that person's institution. 

"'nioy^ supra note 404, at 319 (grant proposals are sent to *the set of colleagues who can most adequately evaluate the proposal but 
who also could use this same information in their own research.*) 

^^ee 1976 House Hearings Report, supra note 3, at 205 ('the most substantial 'rewards* for reviewing are those universally 
regarded as unethical or degenerate: thei\ or plagiarism of ideas, inside information thai provides an advantage over colleagues, and a 
chance to forestall competitors or settle old scores.*). 

"^'Chubin and Hackett observe: 

In the worst case, the pool of appropriate reviewers may shrink to include only those likely to have a conflict of interest 
because they are so close to the proposer as collaborators or colleagues). Selecting reviewers from outside this pool 
would invite an incompetent, incomplete, or inappropriate review, yet within the circle of friendships, rivalries, and 
professional relations jeopardize the character and quality of reviews. Exacerbating these pressures is the shrinking pool 
of real dollars for research . . ., the increasing complexity of science, and the magnitude of competition for resources. 
The ethically difficult task of providing en objective, disinterested, reasoned review may soon be practically impossible. 

Chubin & Hackett, supra note 3, at 80. 
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a financial stake if any qualified judge would have the same stake in the decision/^ Occasions for 
the invocation of the doctrine of necessity are exceedingly rare in the legal world, and they should be 
equally rare in the scientific and fine arts worlds. It is difficult to imagine that there are many 
legitimate fields of scientific inquiry or artistic expression that are so specialized that the only persons 
capable of understanding and evaluating grant proposals are current or prospective grantees of the 
same program.-^ In general, a person with a pending grant application or potential future application 
from the same pool of funds ought not sit in judgment over his or her competitors. 

When information gleaned from the peer review process can be used to the appropriator's 
monetary advantage, the situation poses a clear financial conflict of interest. Even when monetary 
gain is not at issue, use of the peer review process to expropriate another's ideas threatens the 
integrity of the entire peer review-based grants process. Stealing another's ideas plainly constitutes a 
form of plagiarism, a brand of scientific misconduct that has found its way into the popular press 
with some frequency in recent years.*" But it is an especially difficult form of plagiarism to detect 
and prove. 

Applicants are likely to be more circumspect in sharing information with reviewers if they know 
that it may be used to advance the reviewer's reputation or, worse, used to deprive the applicant of 
the recognition that rightfully belongs to the applicant.'*^ At the extreme, the researcher may decline 
to apply for grant funds until the project to be funded is nearing completion to ensure that any 
reviewers do not have an opportunity to get a "leg up" on the applicant.*" Unlike financial conflict 
of interest, which is the subject of detailed regulations in many granting agencies, research conflict of 
interest is not explicitly regulated or even strongly discouraged. Of the agencies studied for this 
report, only EPA formally addressed this sort of conflict of interest in its guidelines, which provide 
that "no Panelists will disclose or use to their own advantage any data, concept, research protocol, or 
any other idea included in the applications."*^ The guidelines do not, however, say what the 
consequences of disclosure or use will be. 

E. Lobbying and Political Pressure 

One very clear threat to the objectivity of the decisionmaking process is its susceptibility to ex 
pane lobbying by interested persons outside of the formal channels of communications. Not only is 
the ex parte attempt to bend the ear of the decisionmaker unfair to those applicants that cannot make 
equal claims to the decisionmaker's attention, but it also gives the appearance that the outcome of the 



*^For example, the doctrine of neceBsily applied when several federal judges challenged the failure of Congress to give all federal 
judges a raise. Since no federal judge lacked a financial conflict-of-interest, any judge could hear the case. Atkins v. United Stales, 556 
F.2d 1028, 1036 (Ct. CI. 1977). cert, denied, 434 U.S. 1009 (1978). 

*^A frequent critic of the peer review system, Rustrum Roy, challenges the proposition that peer review must be narrowly limited to 
persons doing the same work as the applicant. The resuh is that: 

In the present system, the simplest precautions against conflict of interest are ignored. The system flies in the face of the most 
elementary knowledge of human nature and presupposes a level of objectivity, disinterestedness and honesty, such as never obuined in any 
human group. 

Roy, supra note 404, at 318-19. Roy suggesu that a better deflnition of peer is person of equal 'rank* and 'experience* in science, 
*drawn not only from the narrow specialty, but explicitly including neighboring fields.* Id. at 318. 

*"For example, in 1989 the National Institutes of Health accused a prominent researcher at the Baylor School of Medicine of 
appropriating ideas from a manuscript that he was asked to evaluate as part of a pre-publication peer review for a scientific periodical. 
Michael Specter, NIH Accuses Biologist of Stealing Ideas from Rival Researcher. Washington Post, July 13, 1989, A16. 

*^See Darryl E. Chubin, Open Science and Closed Science: Tradeoffs in a Democracy. 10 Science, Technology &. Human Values 73- 
81 (Spring 1985) ('Because peers are usually defined by substantive competence, those playing the above roles are competitors. Every 
incentive exists to retain information that might benefit the other. Whereas the communal spirit would suggest 'we're all in this together,' 
peers are just as apt to set aside ahruism and act 'selflshly,' i.e., to withhold information, at least temporarily, so that some competitive 
advantage, however transitory, is not lost.'). 

*^oy, supra note 404. at 319. 

^^Orientation Handbook, supra note at 181, at 5. 
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process depends more on influence-peddling than upon the merits of the applications/^ If ex parte 
lobbying plays a role in determining who gets discretionary grants, potential applicants without 
"contacts" within the granting agency or the peer review panels will be discouraged from applying, 
and the quality of applications will ultimately suffer.'*^ Very similar considerations apply to attempts 
to pressure the decisionmaker by bringing in influential people from the Administration or Congress. 
Altfiough high level administration officials and congresspersons clearly have a legitimate interest in 
how the agency goes about distributing government dollars, an objective process must proceed on the 
basis of objective criteria, rather than anticipation of legislative favors or fear of congressional 
retaliation. Fortunately, there is little indication that ex parte lobbying, influence peddling and 
outside political pressure play a significant role at any of the scientific institutions studied in this 
report. 

F. Conclusions 

Despite the relative dearth of empirical support for definitive conclusions about the extent to 
which existing peer review grant processes have actually produced biased results, the system as 
currently implemented in the agencies studied here clearly does have a potential for systematic bias. 
Perhaps more importantly, the potential for bias has occasionally given rise to a public perception, 
stronger for some agencies than others, that bias infects the peer review process. While the potential 
for bias identified above is clearly not so great as to warrant abandonment of the peer review process 
or even dramatic changes, it may justify examining proposals for some modest improvements to the 
existing system. 



^^For example few would hold out the discretionary grant process at the Department of Housing and Urban I>evelopment during the 
mid-1980s, where successful applicants had to secure the services of former high level administration ofTiciais to gain access lo the 'real* 
decisionmaking process, as a model for distributing government money in an objective fashion. See generally, Irving Welfeld, HUD 
Scandals: Howling Headlines and Silent Fiascoes (1992). 

^^^When universities began to bypass the peer review process at the major scientific granting agencies in the late 1980s lo lobby 
Congress for direct appropriations for research projects, several prominent representatives of the academic research community passed 
resolutions opposing such 'pork barrel* funding of scientific research. See Donald N. Langenberg, Earmarked Appropriations: The 
I>ebale over the Method of Federal Funding, 20 U. Mich. J. L. Ref. 1029 (1987); High Tech Goes into the Prok Barrel as Congress Helps 
Old Homestate U., 19 Nat. J. 1350 (1987). 
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VII. CONCLUSIONS AND RECOMMENDATIONS 

The peer review model that has evolved into the primary vehicle for awarding discretionary 
grants in the scientific community has proven remarkably successful and durable. The proof is in the 
pudding—the United States is a world leader in most areas of pure scientific research. Government- 
funded research has produced a storehouse of new knowledge, and past governmental expenditures 
have created an infrastructure of educated professionals and physical research capacity that should 
serve as a launching pad for still further scientific advances. The primary reason for this success is 
the intense dedication of the scientists who devote thousands of poorly compensated hours to 
reviewing grant applications and sitting through seemingly endless peer review committee 
meetings.^ More important than the dedication of these volunteers is their integrity, which stems 
largely from a sense of responsibility to the "scientific community" that nurtures and sustains them. 
However nebulous the notion of a community of scientists is to the man on the street, it is very real 
to the professionals from academia and industry who practice science in their laboratories. A 
consciously biased vote in a peer review panel meeting is not only a lapse in personal integrity, it is a 
betrayal of the larger community of professionals. In words of the Director of one of NSF's large 
programs: "Our whole success is based on our credibility."^" Hence, instances of bribery, 
logrolling, and other forms of chicanery that often accompany public discretionary grants programs 
are virtually unheard of in the context of scientific peer review. 

The peer review tradition is not as strong in the arts, social sciences and other professional 
areas. In the arts, where the public funding is a more recent phenomenon, the sense of community is 
less well-developed, and the criteria for evaluation may not be as objective. Yet because they 
recognize that peer review may be the only viable alternative to oppressive government-imposed 
content restrictions,*^^ members of the arts community are also strong proponents of peer review. 
Although the peer review system seems to be working tolerably well in the arts, it has yielded results 
that have, to a much larger extent than in the sciences, attracted outside criticism from the lay public 
and politically powerful critics. To some extent, this is inherent in the subject matter. 
Congresspersons probably deem themselves more qualified to evaluate the quality of artistic 
proposals than to second guess a group of prestigious scientists.""' 

Despite its obvious strengths, the peer review process has potential weaknesses. Some are 
inherent in the concept of peer review and cannot be changed without changing the character of the 
process. They are, in other words, necessary evils. Other weaknesses can be cured. Fortunately, 
there are several practical solutions to reduce the potential for bias and the eliminate the appearance 
of bias in the peer review process, but they all come with a price attached, either in the form of 
additional resources that must be devoted to the process or in the form of reduced efficiency. 

At the outset, reformers should be wary of attempting to induce too many reforms into a system 
that is working reasonably well in most contexts. One of the mystifying aspects of peer review in the 
sciences is the fact that so many highly trained and extremely busy people are willing to devote so 
much time to the process for so little compensation. While prestige is an important form of 



^^1976 House Hearing RcporU, supra note 3, al 33 (The value of the roughly 100,000 reviews contributed free to the INational 
Science] Foundation by the scientific community each year is thus in the range of S2,000,000 to SlO. 000,000.*). 

Telephone interview with Dr. Bill Harris, Director, Science and Technology Centers, National Science Foundation. Nov. 20, 
1990. a02) 357-9808. 

^^Anthony Lewis, Abroad At Home; Fight the Philistines, New York Times, June 8, 1990, sec. A, p. 31, col. 1 ('When politicians 
get into the business of deciding what is legitimate an. the game is up. That is why the NEA was originally set up in a way that insulated 
grant decisions from political pressure, with peer review committees playing a crucial role.*) 

^^Public opinion polls on peer review in the arts are mixed. In a Gallup/Newsweek poll of 500 randomly selected adulu conducted 
on June 23, 1989, 20 percent of the respondents had no opinion, 58 percent favored allowing expert panels, rather than federal officials, 
award the grants. On the other hand, a Los Angeles Times poll of 2,217 adults on September 20, 1989 indicated thai *among those people 
informed about and supporting government funding, 43 percent preferred the 'public voting on which artists should get funding' to 
selection by a peer group of artists by 'other artists' (26%) or by government experts' (14%).' Gladys Engel Lang & Kun Lang. *Public 
Opinion and the Helms Amendment,* 21 Journal of Arts Management and Law 127, 133-35 (1991) 
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compensation in the academic community, reforms that add to the reviewers' burdens may offer a 
further disincentive to undertake an already not-so-attractive task. According to some staffers in the 
scientific agencies, scientists are becoming less reticent to decline invitations to become peer 
reviewers, and it is therefore becoming more difficult to assemble panels with the right expertise to 
evaluate some applications. These problems may only multiply in the future as more academic 
scientists spend more of their time in lucrative consulting relationships. When the alternative to 
serving on a peer review conmiittee is $200 per hour in consulting fees, prominent scientists may 
tend to opt for the latter alternative. 

After examining the threshold question whether peer review should be abandoned altogether in 
favor of a "strong manager" model of grant allocation, this chapter will explore some possible "patch 
and repair" solutions to the bias problem and make some relatively modest recommendations for 
change. The fact that nearly all of the changes suggested here have already been implemented in one 
or more of the peer review granting agencies studied in connection with this Report suggests that 
most of them should not be unduly burdensome. 

A. Advantages of the Peer Review Model 

The peer review model's greatest strength is its ability to achieve objective assessments of 
competing proposals from highly qualified professionals with expertise in the area of interest.'*^ 
When objective criteria can be articulated and applied by persons with training in the relevant field, 
peer review is an exceedingly effective vehicle for shielding the decisionmaking process from 
extraneous and inappropriate considerations. In addition, because the applicants are being judged by 
their peers (or in many cases their "betters"), they are generally inclined to accept the outcome. 
Indeed, the peer review system is a vehicle for rewarding excellence in the arts and sciences quite 
apart from the monetary support that comes from funding decisions.''" Because it involves the input 
of many reviewers, it is generally not perceived as being autocratic or arbitrary. Finally, peer review 
helps to shield governmental decisionmakers from the political pressures that usually exist when a 
bureaucracy has the power to dole out public dollars. This is especially important in the arts, where 
the pressures to impose a narrow cultural uniformity on government funded art are great. ^'^ 

B. Peer Review versus the "Strong Manager" Model 

The discussion of the peer review process in the foregoing chapters began with the assumption 
that peer review is the most appropriate decisionmaking model for dispersing government funds in 
the programs being described. It is therefore appropriate at the outset to consider the possibility of 
abandoning the process altogether in favor of an alternative model that might more efficiently lead to 
more objective results. 

A host of alternative decisionmaking models exist. For example, funds might be allocated 
through a lottery in which the agency staff would screen all applications for an initial threshold 
showing of competence and relevance and then conduct a lottery to determine which of the 
meritorious proposals are funded. Although this process would be completely free of favoritism, 
animus, tunnel vision, conflict of interest, and outside pressure, it would not fulfill the programmatic 



^^1976 House Hearing Reporu, supra note 3, at 32. 

^^O'Rourke, supra note 312 (Peer review *as one of the few acts of approval and applause in the an world that is out of the nurket 
\oop. beyond the reach of the gatekeepers and taste-makers, those who are involved in the commercialization of art.*) 

^^ee Fiss, supra note 298. Vartan Gregorian, President of Brown University argues that 

Government money is not the government's; it's the people's. It needs checks and balances to be spent wisely, but that is why the 
peer-review panels are there.* 

Glueck, supra note 320. One might legitimately quesiton Gregorian's conclusion thai 'experts' in the arts are the best judges of how 
the 'people's* money should be spent. As Part 5 described, Congress has recently required NEA to appoint at least one lay person to each 
of its peer review panels. See also text accompanying notes 513-523, infra (suggesting that lay participation be expanded on peer review 
panels in all granting agencies). 
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goal of awarding limited monies to the "best" projects.*" In situations in which society can place 
little confidence in the ability of human beings (even with considerable expertise) to select the "best" 
from among a group of otherwise acceptable alternatives, a lottery is an extremely efficient way to 
make the choice.''^ Although this kind of situation probably arises more often in governmental 
decisionmaking than most policymakers willingly acknowledge, scientific research grants is probably 
an area in which society can trust informed experts objectively to pick the best proposals from among 
a pool of good proposals. Whether the same confidence may be placed in the ability of experts to 
distinguish the best art and music from among a host of acceptable proposals is more debatable. At 
this juncture, however. Congress has elected not to fund the arts through a lottery. Despite a great 
deal of recent controversy over public funding to the arts, few would second guess this decision. 

Assuming that it is possible to choose the "best" proposals in an objective way according to 
predetermined criteria, it does not go without saying that the entity making that choice should be a 
group of the applicants' professional peers. The most frequently mentioned alternative to peer review 
is the "strong manager" model in which highly trained government officials pick the winners and 
losers from among the qualified applicants with the aid of objective criteria and/or a formula. This 
model is familiar in the context of government contracts awarded on the basis of competitive bidding. 

Rustrum Roy, a frequent and persistent critic of the peer review system in United States science, 
has suggested a formula-based strong manager model for funding scientific research. Under this 
model, the decisions would be made by permanent agency staff, rather than outside peers, but the 
staffs discretion would be constrained by Roy's suggested formula. The formula is based on the 
principle that grant applications should be submitted by institutional departments, not individual 
researchers, and be based upon the past productivity of the applicant institution in the relevant field. 
A department's productivity would be measured by the number of papers published in agreed-upon 
journals; the number of M.S. and Ph.D. degrees granted; the aggregate research support from all 
mission agencies for the faculty; and total support for research from industry.'"' 

Roy's simple formula for determining the sum to be allotted to a department or interdisciplinary 
laboratory is as follows: 

Total sum to be granted to unit = 

A X (number of publications) + 

B X (weighted number of advance degrees) + 



^^See Chubin & Hackeu, supra note 3, at 198. 

^'^Sec McGarity, Multi-Party Fonim Shopping for Appellate Review of Administrative Action, 129 U. PA. L. Rev. 302 (1980); 28 
U.S.C. §21 12(a). 

^'^ustum Roy, Peer-Reviewed Produclivity-Based Formula for Funding University Research, reprinted in Research Project Selection 
Hearings at 24. 

Dr. Ercole Cavilieri, a cancer researcher and well-known maverick at NCI, suggested a similar model for awarding discretionary 
grants in acience: 

A scientist first learns how to do research in the process of obtaining a Ph.D. degree. If his or her supervising professor 
believes he or she is a strong candidate for funding, the professor nominates the graduate student for a post doctoral 
assignment at the laboratory of another scientist. The student would have a supervisory committee both at graduate 
■chool and at the institution at which he or she had the post doctoral assignment. The supervisory committees would 
have an opportunity to observe both the intellectual capacity of the student and also his or her performance as a 
researcher. After a year or two of post doctoral work, the student would be nominated by his or her graduate school 
supervisor for a research division grant. The conimittees and the professor would be expected to vouch for the student's 
ability to do high quality research. NIH would grant him a 5 year development grant. If the scientist has performed well 
after five years, he or she would be entitled to continuing support for so long as the researcher continues to produce an 
acceptable volume of high quality research. 

Telephone interview with Dr. Ercole Cavalieri, Professor & Researcher at the Univ. of Nebraska Medical Center, Eppley Institute for 
Research in Cancer, Eppley Institute of Pharmaceutical Sciences. Nov. 15, 1990. 



Peer Review in Discretionary Grantmaking 87 

C X (sum received for research from mission-oriemed agencies} + 

D X (sum received for research from private industry) 

The weighting factors - A, B, C, D - would be adjusted by each agency so that the sum of the 
monies distributed to all institutions would be equal to the total budget.** If necessary an element of 
peer review could be incorporated into the system by allowing a panel of peers evaluate the 
credentials and immediate past performance of individuals in the applicant department.*" 

1. Advantages of the Strong Manager Model 

The strong manager model has numerous advantages over the peer review model. First, and 
most obviously, it is less expensive. The resources that are required to assemble peer review panels 
are available for funding research under the strong manager model. Prestigious and highly 
productive professionals, who would otherwise spend time writing proposals and engaged in peer 
review evaluations, could spend more time on their own research and other creative efforts. 
Especially in programs in which a very large number of qualified applicants are competing for a 
small number of grants, it seems wasteful to demand that productive professionals spend large 
amounts of time writing and evaluating proposals, no one of which has a high probability of 
success.*^ One prominent researcher complained to NSF that: 

The amount of time and effort spent on preparing proposals, reviewing them after 
inefficiently short periods, and administering them, is an astronomical cost in terms 
of science that is not getting done. When I observe my colleagues here in the United 
States I am saddened to see creative, trained minds, diverted away from productive 
research into huckstering .... This society is so super-competitive that the process 
of creativity is becoming swamped by constant evaluations. We are forced into 
presenting ourselves better and better, rather than doing more and better science. 
The peer review process has much to recommend it, but I suspect that we are 
overdoing it in a big (expensive) way.*"' 

The strong manager model only requires a staff made up of competent, but not necessarily 
prestigious, professionals who are capable of applying the formula in a consistent manner. 

Second, the strong manager model gives more emphasis to past productivity and less to the 
ability of an applicant to draft an attractive proposal. According to Rustrum Roy, "Only the most 
sanguine advocate unfamiliar with the literature would claim that there is any basis for expecting a 
correlation between a scientist's ability to present an essay and the actual future production of the 
'best science'."**" A National Academy of Sciences study of peer review in NSF did not find a high 
correlation between grants awarded and measures of the previous scientific performance of the 



**'Roy, iupra note 404, at 322. 

**'Roy, gupra note 439. at 24, 40. 

*^Roy, supra note 404, at 318. Roy uses the example of a decision in 1983 by the I^panmeni of E>efen8e to allocate $30 million 
pursuant to research proposals from universities. This resulted in 2,200 proposals totalling S625 million. The peer review panel 
assembled to make recommendations about the applications necessarily spent a great deal of time evaluating proposals that ultimately could 
not be funded. Id. at 317. 

**^eril Review Task Forec, supra note 100, ai 5 (the report does not identify the commenier). 

***Roy, supra note 404, at 318. See also Roy, supra note 6, at 48. The recent report of the NSF Merit Review Task Force suggested 
that more attention be devoted to an applicant's past track record. Merit Review Task Force, supra note 100, ai 15-16. The report cited 
favorably the model of the Natural Sciences and Engineering Research Council of Canada, in which proposals from established 
investigators focus more on the recent track record, while new investigators follow a different proposal format that emphasizes the potential 
for creative work. 
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applicants, and there was some indication that chance played a large role in grant awards under the 
peer review model. ""^ 

Third, the strong manager model eliminates the potential for research conflict of interest that 
exists in the peer review model. An applicant's competitors are not in a position to affect the 
outcomes of the applicant's proposals. Moreover, until the agency has made a final decision whether 
or not to fund a project, the applicant's ideas need not be disclosed to anyone other than the agency 
staff. 

Fourth, the strong manager model can provide the continuity necessary for a sustained and 
successful research program. Institutions that produce high quality research on a continuous basis 
can be assured of continued funding over the long haul.*^ Researchers can pursue "chance leads" 
that arise in their research without fear that they will stray too far from the project that was funded.**' 

Fifth, the strong manager model is appropriate for granting agencies with clearly defined 
missions that may vary from the policy preferences of the reviewers, who might become afflicted 
with tunnel vision. In such agencies, "the staff member can better judge the key issue of the 
importance of a proposed bit of research in fulfilling the mission's objective than can extramural 
experts."** Roy points out that proposal peer review is not used by mission-oriented agencies like 
the Department of Defense, NASA, the Bureau of Mines, or the Department of Energy. In 
particular, he notes that the highly successful Office of Naval Research and the Defense Advanced 
Research Projects Agency have never used peer review and continue to use a strong manager 
system.**' 

In sum, the strong manager model institutionalizes the "halo effect." The most heavily 
credentialed researchers would tend to get funding, whether or not they could write persuasive 
proposals. There would be an effective entitlement to funding until such time as the researcher 
demonstrated that he could no longer perform good research. 

2. Disadvantages of the Strong Manager Model 

Perhaps the greatest danger of the strong manage- model is the very real possibility that agencies 
will award money for bad science to support a crony of upper level appointees or to advance a 
particular politicaJ perspective. This problem, which is sometimes referred to as "staff abuse,"*" 
seems endemic to grant procedures in which great discretion is placed in the hands of a few relatively 
unaccountable low-level governmental officials. EPA's experience in the early 1980s and the 
experience of the Department of Housing and Urban Development with discretionary grants in the 
mid-1980s should give pause to anyone who would advocate substituting politically appointed agency 
staff for peer reviewers.**' 

Because the strong manager model focuses heavily upon past accomplishments, it would tend to 
be biased against newcomers to a scientific field. Roy recognizes this potential and would shield 
against it by allowing any unit that adds new faculty members to receive one or more extra credits 



***Cole, Cole and Simon, (upra nole 126, at 885. The study concluded that approxiniately iwenty-rtvc percent of panel decisions 
would be reversed by a different panel. 'Since the reversal rate is about 25%. we nuy conclude thai the fate of a particular grant 
application is roughly half determined by the characteristics of the proposal and the principal investigator, and about half by apparently 
random events which might be characterized as the 'luck of the reviewer draw." Id. See also Alan H. Clark, loick. Merit and Peer 
Review, 215 Science 1 1 (1982) (arguing that this fact does not justify eliminating the peer review nunlel for awarding grants). 

According to Roy: *A great advantage of the (strong manager) system is thai no siep-funciion changes up or down are possible, 
preserving the continuity essential in long-term basic research.* Roy. supra nou 439. at 24. 

*"Roy, supra note 6. 

**1976 House Hearing Reports, supra note 3, at 39. 

**'Roy, supra note 6, at 47. For a brief description of the granting system at the Office of Naval Research in the 19708, see 1976 
House Hearing Reports, supra note 3, at 20. 

4501976 House Hearing Reports, supra note 3,ai 25. 
'See generally, Welfeld, supra note 428. 
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equal to the average research support for each faculty in the unit.**- This does not, however, allow 
for the institution that desires to achieve prominence by building from the ground up, and it would 
make it very difficult to shift ftinding out of departments that become moribund. 

Institutionalizing the halo effect through the strong manager model would also tend to increase 
the disparity between "have" and "have not" institutions, because the best researchers tend to locate 
at the most prestigious institutions.*" As a result one institution might well have several well-funded 
departments, while another received no funding at all. Over time this would become self-reinforcing, 
as faculty members with high potential gravitated to the departments that were most securely funded. 

The efficiency advantages of the strong manager model should not be overemphasized. While 
awarding a smaller number of multi-year grants to a few departments would reduce the amount of 
resources devoted to the grant awards process at the front end, it might result in the adoption of more 
protective procedures at the back end when the granting agency sought to deprive an existing 
department of its grant at renewal time. The grants, in other words, night come to more closely 
resemble entitlements. 

Finally, by placing heavy reliance upon publications, Roy's formula-based version of the strong 
manager model would to some extent simply shift the locus of peer review from the grants process to 
the private sector journal peer review process where governmental privacy and due process 
protections are not applicable and where there may be even fewer protections against bias.*** 

3. Conclusions 

The case for wholesale abandonment of the peer review model for discretionary grants in the arts 
and sciences is far from compelling. In the main, peer review seems to be functioning quite 
efficiently, and the occasional criticisms of the process in practice do not appear to warrant rejection 
of the model. 

Peer review works best in cautious programs in which incremental gains are expected and where 
accuracy is a paramount goal. Because it tends to result in bias against mavericks, however, it can be 
counterproductive in programs designed to explore fresh ideas and innovative approaches. Agencies 
that administer programs aimed at enhancing innovation should consider reducing or eliminating the 
role that peer review plays in awarding discretionary grants. One intermediate solution, therefore, 
would be to emulate NSF's recently adopted "expedited awards" program in which a small 
percentage of each Program's budget can be set aside for small grants of limited amount and duration 
without peer review. If the seed grants bear fruit, the mavericks should have a better chance to prove 
the merits of their ideas to skeptical peer reviewers in the next funding cycle. In 1989, an NSF 
appointed panel examined the recently implemented program and concluded that it was an excellent 
vehicle for funding "untested and novel ideas, ventures into emerging research areas, new expertise 
and approaches to 'old' topics, (and] new multi-disciplinary work."*^* 



*^^Roy, supra note 6, ai 79. 
^^See Chubin & Hackett, supra nolc 3, at 42. 
*^ee Chubin & Hackett, supra note 3, at 201 . 
^^Quoied in Bjerklie, supra note 105. 



90 Thomas O. McGarity 



RECOMMENDATION 

Agencies that rely upon peer review to evaluate grant proposals should consider setting aside a 
small portion of the available funds for awards to innovative proposals out of the mainstream based 
upon recommendations of the staff without peer review. Such awards should be of relatively brief 
duration, and should be subject to renewal only through the ordinary peer review process. 

A. Openness and Accountability as Shields Against Bias 

Bias can flourish in closed peer review systems in which panel meetings are conducted in 
private, mail reviews are unavailable for scrutiny and rebuttal by rejected applicants, and the relevant 
economic and research interests of the reviewers are not disclosed. By its very nature, peer review 
tends to be secretive and unaccountable to applicants and the general public. Two long-time students 
of the peer review process observe that: 

Peer review is an intensely private process that originates with a scientist's mind, 
continues on paper as a bureaucratic procedure and ends behind the closed doors of a 
funding agency. The process is at nearly all points inaccessible, opaque, and heavily 
infused with the values and interests of stake holders. Peer review leaves few clues 
in the public domain, and many participants in the system insist on minimizing public 
access to information.*^ 

On the theory that "sunlight is ... the best disinfectant,"*^^ one relatively modest protection 
against bias in peer review is to open the process up to greater scrutiny by affected applicants and the 
public. The House subcommittee that studied peer review in NSF in 1976 recommended that NSF 
attempt to achieve "[t]he greatest degree of openness [in the] award-decision making process 
consonant with effective proposal evaluation and reasonable efficiency ...."*** 

Parts 2-5 revealed that the peer review systems in NIH, EPA and NSF have evolved in the 
direction of greater openness and accountability, but some agencies have evolved more rapidly than 
others. The evolution in the scientific agencies toward greater opennes'^^ has mirrored a general 
opening up of the bureaucratic process through the enactment of the Freedom of Information Act 
(FOIA),*^ the Federal Advisory Committee Act (FACA),**° and the Privacy Act,**' all three of which 
apply directly to various aspects of the peer review process in the granting agencies. 

1. The Existing Legal Regime— The Freedom of Information Act, the Federal Advisory 
Committee Act, and the Privacy Act 

The FOIA requires every federal agency to make available to any person any record in the 
agency's possession upon a request by that person that reasonably describes the record. This broad 
requirement, however, has several important exceptions. The exemptions most relevant to peer 
review in granting agencies are exemptions (4), (5) and (6). Exemption (4) includes "trade secrets 
and commercial or financial information obtained from a person and privileged or confidential," and 
exemption (5) covers "inter-agency or intra-agency memorandums or letters which would not be 
available by law to a party other than an agency in litigation with the agency." Exemption (6) applies 



*^*Chubin & Hackett, supra note 3, at 50. 

*^^Louis D. Brandeis, Other People's Money 92 (1914) ('Sunlight is said to be the best disinfectant; electric light the most eflicient 
policenian*). 

**1976 House Hearing Reports, supra note 3, at 5. 
*^5 U.S.C. §552. 
*®5 U.S.C. App. 2. 
**"5 U.S.C. §552b. 
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to "personnel and medical files and similar files the disclosure of which would constitute a clearly 
unwarranted invasion of personal privacy. "'^^ 

The FACA requires federal agencies that rely upon recommendations of advisory committees to 
charter those committees.*" The term "advisory committee" is defined functionally to include any 
committees that are "utilized" by federal agencies "in the interest of obtaining advice or 
recommendations for" those agencies. Peer review committees clearly come within this broad 
definition.*** Each advisory committee's charter must set out the committee's objectives, duties, 
number and frequency of meetings, and termination date. Insofar as the Government in the Sunshine 
Act does not provide otherwise,*" public notice must be given of all advisory committee meetings, 
they must be "open to the public," and interested persons must be permitted to attend and file 
statements.** The Government in the Sunshine Act allows an agency to close meetings or portions of 
meetings where an open meeting might result in disclosure of, inter alia, trade secret and 
commercial or financial information that is privileged or confidential, or information "of a personal 
nature where disclosure would constitute a clearly unwarranted invasion of personal privacy."*" The 
agency must prepare minutes for advisory committee meetings and make those minutes available to 
the public, subject to the exemptions in the Freedom of Information Act.** Transcripts of advisory 
committee meetings must be made available to the public, subject to the same exemptions.** 

The Privacy Act requires federal agencies to protect personal information in agency files from 
unauthorized disclosure, to publish descriptions of the existence and nature of the records containing 
personal information about people, and to give individuals access to review and copy information 
about themselves and to demand that the agency correct any information that is not accurate, relevant, 
complete or timely.""" The agency must either make the requested correction or inform the individual 
of its reasons for failing to do so. The agency must give an individual whose request is denied a 
right to review that decision within 30 days."^' The Act, however, only applies to "systems of 
records" from which records are retrieved by the name of an individual or other personal identifier. 
This latter qualification has proven controversial and difficult to interpret. The Privacy Act also 
contains several exemptions, one of which is for "investigatory material compiled solely for the 
purpose of determining suitability, eligibility, or qualifications for Federal civilian employment [or] 
Federal contracts," but only "to the extent that the disclosure of such material would reveal the 
identity of a source who furnished information to the government under an express promise that the 
identity of the source would be held in confidence."*^ 

2. Reviewer Meetings 

Most peer review granting agencies facilitate openness and accountability by holding open 
committee meetings. Open meetings allow outsiders to observe the decisionmaking process in action. 
Even when meetings are legitimately closed to the public to protect privacy interests or to preserve 
candor, each committee member has the opportunity to observe the demeanor of the others and to 



*^ U.S.C. 8552(b)(4). (5), (6). 

*^5 U.S.C. App. 2. §9(c). See also Uniled States General Accounting OfTicc, Peer Review: Compliance With the Privacy Act and 
Federal Advisory Committee Act (1991) (reporting that most agencies studied in the report chartered peer review committees as advisory 
conrunittees under FACA). 

'^ee Memorandum Opinion for the General Counsel, National Endowment for the Humanities, from Leon Ulnun, Deputy Assisuni 
Attorney General, Office of I^gal Counsel, August 18, 1980 (mimeo, copy on file with author). 

*"5 U.S.C. App. 2 §IO(d). 

**5 U.S.C.. App. 2 110(a)(1), (3). 

*^S U.S.C. §552b(c)(3). (4). (6). 

**5 U.S.C. App. 2 §10(b). (c). 

**'5 U.S.C. App. 2 §1 1(a). 

'"^S U.S.C. §552a(b). (d), (e). 

""5 U.S.C. §552a(d)(3). The review period may be extended another 39 days for good cause. 

^"^ U.S.C. §552a(k)(5). 
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challenge their evaluations. A committee member who would play favorites or blackball an applicant 
must be prepared to meet the honest inquiries of his or her fellow experts on the committee. Hence, 
bringing all of the reviewers together to discuss their opinions can in itself be a powerful shield 
against favoritism and animus.*'^ All of the granting agencies studied in this report provide for peer 
review panels, and some assemble more than one group of peers for a single round of evaluations.^'* 

The principal disadvantage of panel meetings is the expense, both in terms of the resources 
required to bring the experts together and the valuable time of productive professionals consumed in 
travel and meetings. Meetings are almost always less efficient than one-on-one communications. In 
addition, since it will usually be more difficult for upper level decisionmakers to reach results that 
differ greatly from a panel's unanimous recommendations, meetings can reduce the discretion of 
politically accountable decisionmakers.*'* Finally, meetings can facilitate old boyism and the halo 
effect, and they will not necessarily limit any tendency of peer reviewers to downgrade mavericks. 

Nevertheless, the effort seems warranted in most cases. Where expense is a genuine constraint, 
alternatives exist to facilitate peer review meetings at very little overall cost. For example, it should 
be relatively inexpensive in the context of mail reviews at NSF to circulate written reviews to each of 
the reviewers and conduct a single conference telephone call to discuss and possibly amend the 
reviews. Such an informal meeting could greatly enlighten the agency staffer who must rank the 
proposals for upper level decisionmakers. It could provide valuable feedback to the reviewers as to 
how their peers evaluated the same proposals. 

Assembling the reviewers for a meeting will usually bring the process within constraints of the 
Federal Advisory Committee Act. With several important exceptions, meetings subject to FACA 
must be open to the public. Although it may be difficult to open telephone conference calls to the 
general public, it may be feasible to invite interested members of the public to listen to the contents 
of the conversations either in person or on tape. The extent to which the meetings should be open to 
the public and minutes and transcripts be made available to applicants and the public will be 
discussed below. The essential point here is that there is virtue in meetings, whether or not their 
contents are open to applicants and/or the public. 

RECOMMENDATION 

Agencies that rely upon peer review to evaluate grant proposals should generally assemble the 
reviewers for a meeting in which each reviewer has an opportunity to comment upon the evaluations 
of the other members in the presence of the other members. Such meetings may be arranged in 
person or by telephone conference call. 

A. Feedback and Rebuttal 

The decision not to fund a grant application can have serious consequences for an academic 
researcher's career. Younger scientists at prestigious institutions have no hope of becoming tenured 
if they do not have at least one NIH or NSF grant. The denial of a grant application to a more 
established researcher can result in the closing of his laboratory and effectively end his or her career 
as a productive researcher.*'* For some artists and musicians, NEA grants are a principal source of 
livelihood. A rejected applicant in any of these agencies understandably wants to know why the 
funding agency took that potentially career ending action. If the applicant discovers that a mistake 



^'^1976 House Hearing Reports, supra noic 3, ai 37-38. 

*'*NSF relies upon mail reviews without meetings for many of its programs, but the sufT always has the option to assemble the ouuide 
reviewere for a meeting to discuss all of the proposals. 

*'*See 1976 House Hearing Reports, supra note 3, at 38. 

*'^ee Kolata, supra note 18 (reporting statements made at a National Academy of Sciences Forum on the crisis in research funding); 
Booth, Biomedical Scientists Cite Funding "Crisis," Washington Post, June 29, 1990, A25, coi 1 (same). 
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was made in the evaluation, fairness demands that the applicant be given an opportunity to correct the 
error and to rebut the decision not to fund before the final funding decision is made. 

1. Feedback 

One of the most powerful procedural protections against bias is the requirement that the 
decisionmaker state his or her reasons. '•^ Just as judges in the legal system are generally obliged to 
provide reasons for the exercise of judicial power, peer reviewers could be required to provide a 
statement of reasons to rejected applicants. A reasons requirement is, however, not a panacea. Any 
sophisticated observer of the legal and administrative process knows that an agency's statement of 
reasons can be a post hoc rationalization for a decision reached upon unarticulated or improper 
grounds. It is often easy enough for the decisionmaker to find the bottom line first and then come up 
with a rationale to explain that result. Still, forcing the peer reviewers to state reasons can enhance 
the acceptability of the decisions among the pool of rejected applicants and the reasons can help them 
improve future applications."^ 

The FOIA and the FACA require granting agencies to provide some feedback to rejected 
applicants as members of the general public. The Privacy Act goes beyond FOIA and FACA to 
make particular information about individuals available to those individuals even when that 
information need not be disclosed to the general public. Thus, the question whether feedback is 
appropriate has to some extent already been decided by Congress, and the granting agencies are 
obliged to follow the statutory directives. 

The Privacy Act requires federal agencies that maintain a system of records in which the records 
of individuals are identifiable to allow individuals to review and copy information about themselves, 
and FOIA similarly requires agencies to provide copies of agency documents to any person. But 
neither statute requires that the agency generate any written information that it would not otherwise 
prepare; nor do they require the agency to maintain information in the applicants' files for any 
particular length of time. Thus, for example, an applicant for an EPA grant is theoretically allowed 
to see the written evaluations of individual reviewers, but the evaluations are in the agency's file only 
for as long as it takes the agency staffers to incorporate their substance into a "Summary Statement." 
Thus, as a practical matter, the applicant does not have access to the written reviews. 

The FACA gives applicants the right to observe peer review committee meetings where outside 
evaluations are discussed, and it further requires that all "records, reports, transcripts, . . . working 
papers, . . . studies, ... or other documents which were made available to or prepared for" advisory 
committees be made available for public inspection.'*^ But FACA also allows agencies to close 
advisory committee meetings to the public when necessary to protect trade secrecy and personal 
privacy, and the right to inspect documents is subject to the exemptions in FOIA. 

The public availability of committee working papers, including initial peer reviews, under FOIA 
raises several complex legal issues, some of which have been answered in the seminal case of 
Washington Research Project, Inc. v. Dept. of Health, Education and Welfare.^ In that case an 
organization sought information on eleven projects that NIH had funded through the National 
Institute of Mental Health to study the effects of psychotropic drugs on the behavior of children with 
certain learning disabilities. In particular, they sought the grant applications and information 
collected pursuant to the peer reviews. At that time NIH only publicized a list of research grants that 



"'^Scc, e.g., Enlemalional Harvester Co. v. Ruckelshaus, 478 F.2d 615 (D.C. Cir. 1973); Friendly, Some Kind of Hearing, 123 U. 
Pa. L. Rev. 1267. 1292 (1975). 

"'^After conducting extensive hearings into the peer review at NSF in 1076, a House subcommitice recommended: 

The Foundation should include in each letter announcing a funding decision to an applicant either a statement of the methods and 
raliorule of the decision or a statement that such information will be provided on request. 
1976 House Hearing Reports, supra note 3. 

^"^5 U.S.C. App. 2 §10(b). 

^504 F.2d 238 (D.C. Cir. 1974). cert, denied, 421 U.S. 963 (1975). 
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were awarded, including a general description of the project and its budget. The agency claimed that 
aJl of the requested information came within exemptions (4), (5) and (6) of FOIA. 

The court of appeals first rejected the agency's argument that a grant application was protected 
from disclosure by the trade secret exemption (exemption (4)), noting that the exemption was "not 
necessarily coextensive with the existence of competition in any form."^' Thus, despite the fact that 
the ideas contained in research grant proposals were the researcher's "stock in trade," they were not 
protected, because they were not commercial. The court concluded: 

It is clear enough that a non-commercial scientist's research design is not literally a 
. trade secret or item of commercial information, for it defies common sense to pretend 
that the scientist is engaged in trade or commerce.^ 

In a footnote, the court noted that NIH regulations at that time precluded grant awards to for- 
profit institutions.^ 

The court held, however, that the pink slips and site visit reports (and presumably reviewer 
reports to peer review panels) were protected by exemption (5), the exemption for internal agency 
documents. Although a peer review panel was not itself a federal agency, it was "performing staff 
functions through the medium of outside consultancy.*^ Thus, while exemption (5) did not shield 
purely factual material, it did allow an agency to reflise to disclose "materials reflecting deliberative 
or policy-making processes."*" Even purely factual matter could be exempt if it was "inextricable 
without compromise of the deliberative process."^ Similarly, a summary of factual information that 
was part of the deliberative process could be withheld. Applying these statutory principles, the court 
concluded that most of the material in the site visits and pink sheets did not have to be disclosed. In 
particular, the factual material in the pink sheets was sufficiently connected to the deliberative 
process so as to preclude disclosure of any information contained therein. 

FACA does not have a similar "internal deliberations" exemption, because the whole purpose of 
FACA is to open up such deliberations to the public. Thus the primary rationale for declining to 
make peer review evaluations available to requesters under FOIA is unavailable to requests under 
FACA to attend peer review committee meetings and to have access to materials relied upon and 
discussed at those meetings. The Washington Research Project court's resolution of the trade secrecy 
question would seem to govern both FOIA and FACA requests, and thus preclude that rationale for 
limiting public access to peer review documents. 

Recent developments, however, may have undermined the Washington Research Projea holding 
on the trade secrecy issue. NIH has changed its policy of not funding private research enterprises, 
and some academic scientists have become wealthy as a result of research programs initially funded 
by NIH. In the booming area of biotechnology, lucrative pannerships and consultantships are no 
longer the exception to the rule that academic scientists are devoted, but poorly compensated, seekers 
of scientific truth. In short, it no longer "defies common sense to pretend that the scientist is 
engaged in trade or commerce." Yet even if the trade secrecy exemption may be applicable to some 
grant applications in the area of biotechnology, the Washington Research Project rationale would 
still seem to preclude using the trade secrecy exemption to shield from public disclosure peer review 
information about applications in other hard sciences, the social sciences and the arts. 

The FACA, however, has an additional exemption for advisory committee consideration of 
matters for which meetings may be closed under the Government in the Sunshine Act— viz.. 
information "of a personal nature where disclosure would constitute a clearly unwarranted invasion of 

*'504 F.2d at 244. 

*^-lcl. 

*^504 F.2d8t244. n. 6. 

**504F.2dat246. 

*"504 F.2d at 249. quoling Environmental Protection Agency v. Mink. 410 U.S. 73. 89 (1973) 

^504 F.2d at 249. 
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personal privacy." This exemption is considerably broader than FOIA's exemption for "personnel 
and medical files and similar files the disclosure of which would constitute a clearly unwarranted 
invasion of personal privacy." Arguably, peer review evaluations are "of a personal nature" to both 
the reviewer and the applicant. A reviewer's comments are personal in the sense that they do not 
reflect the ideas or assessments of other persons. On the other hand, the peer reviewers are 
constantly urged to evaluate the applications in their professional roles without regard to personal 
considerations, like the reviewer's friendship with the applicant.-**^ Similarly, career-oriented 
information is generally considered professional, rather than personal, from the perspective of the 
applicant. If a reviewer's comments are of a personal nature, then the agency could reasonably 
conclude that disclosing them to the public would constitute a clearly unwarranted invasion of 
personal privacy. On the other hand, it is not obvious how disclosing the comments of other persons 
on the applicant's proposal constitutes a clearly unwarranted invasion of the applicant's personal 
privacy.^ While the case for closing peer review panel meetings to the public under the FACA is 
plausible, it is not especially compelling.*^ 

Assuming that the granting agencies can shield most peer review information from public view, 
the Privacy Act may still require them to share such information with the applicants themselves. 
That statute requires agencies to give individuals access to review and copy information about 
themselves in agency files. The Privacy Act lacks the trade secrecy, internal agency deliberation and 
personal privacy exemptions, but it contains at least one exemption that may be applicable to peer 
review evaluations. Agencies may promulgate rules to exempt records that constitute "investigatory 
material compiled solely for the purpose of determining suitability, eligibility, or qualifications for 
civilian employment or federal contracts to the extent that the disclosure of such material would 
reveal the identity of a source who furnished information to the government under an express promise 
that the identity of the source would be held in confidence. " 

This Privacy Act exemption would appear to be applicable to peer review documents concerning 
discretionary grant applications if the word "contracts" includes discretionary grants. Although the 
subject of great debate between the executive branch and Congress, this question has not been 
resolved by a court. The American Law Division of the Congressional Research Service has 



^^See Memorandum Opinion, supra nole 464 (mimeo. on Hie with author) (*to invoke a Sunshine Act exemption, a more specific 
justification must be found to exist than merely a generalized need to protect candor in advisory committee deliberations*). 

^^A memorandum prepared by the General Counsel of the National Endowment for the Arts found that resolution of the issue 
involved a balancing analysis: 

The subjects to be discussed with respect to applications for financial assisunce could well include, for example, an 
applicant's abilities in his field, his reputation among his colleagues, and his professional background in performance. 
These topics would certainly appear to involve the type of pereonal information in which an applicant has a privacy 
interest. But the fact that an applicant has a legitimate privacy interest in a closed committee meeting does not end the 
inquiry. The agency must also determine that the privacy interest is not de minimis and is not outweighed by 
counlerveiling interests in openness. 

Memorandum Opinion, supra note 464 (mimeo, on file with author). 

^'^rhe legislative history of the addition of the Sunshine Act exemptions to the Federal Advisory Conimittee Act is not especially 
enlightening. The Conference Committee Report speaks directly to the issue of peer review panels, but does not resolve it: 

The conferees . . . are concerned about the possible effect of this amendment upon the peer review and clinical trial preliminary 
review systenu of the National Institutes of Health. The conferees thus wish to suic as clearly as possible that pereonal dau, such as 
individual medical information, is especially sensitive and should be given appropriate protection to prevent cleariy unwarranted invasions 
of individual privacy. Although the conferees are sympathetic to the concerns expressed by NIH ... the conferees are equally 
sympathetic to concerns expressed by citizen's groups that important fiscal and health-related information not be unnecessarily withheld 
from the public . 

H.R. Conf. Rep. No. 1441. 94th Cong.. 2d Sess. 26 (1976). 
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concluded that the word "contract" in the exemption does not include discretionary grants,*" but the 
General Counsel of NSF has concluded that it does.''" 

NSF takes the position that federal grants and awards are a subset of the broader category of 
federal contracts and that in the context of grants, the terms are interchangeable. CRS argues that 
contracts and grants are distinct entities. Contracts are limited to civilian and military personnel 
employment and traditional procurement contexts, whereas grants involve less supervised dispersals 
of government largess without a well-defined quid pro quo. The statute is at best ambiguous, and 
each side of the debate can cite ample legislative history to support its view. Ultimately, the matter 
will have to be resolved definitively by a court, or Congress will have to resolve it with new 
legislation. 

All of the granting agencies studied for this report close peer review committee meetings during 
the discussion and ranking of applications.*^ As we have seen, despite the Washington Research 
Project holding that pink sheets need not be disclosed to the public, NIH routinely makes these staff- 
prepared summaries of the comments of peer reviewers available to applicants, and if pressed it will 
provide rejected applicants with redacted copies of study section minutes. Although NSF originally 
failed to provide reasons to applicants, its recent procedural changes should ensure that most 
applicants learn the reasons for the reviewers' recommendations. EPA provides summary statements 
to rejected applicants, and they are theoretically entitled to see the written comments of the primary 
reviewers, but since these are discarded as soon as the summary statements are drafted, applicants do 
not as a practical matter have access to them. The practice in NEA varies. Some Program 
Specialists are willing to describe in detail the considerations that motivated the panelists; others only 
send out a form letter containing the applicant's score. 

As we observed in Part 6, a reasons requirement is a powerful legitimator of a decisionmaking 
process, and it usually is relatively cost-free, because the system itself usually generates written 
statements of reasons, albeit sometimes in rather primitive form, for its own internal purposes. 
Granting agencies that rely upon the peer review model should go beyond the minimum requirements 
of FACA and the Privacy Act to adopt NIH's practice of preparing detailed one-page summary 
statements of panel evaluations on a routine basis, whether or not an applicant requests that 
information. Although this requires some additional staff time, it should not be excessively 
burdensome. Some documentation of panel deliberations is no doubt undertaken in any event. 
Reducing those notes to a one page explanation should not add greatly to the staff time already 
devoted to the process. In addition, granting agencies should make any written evaluations prepared 
by the peer reviewers and minutes of any peer review panel meetings available to applicants as 
requested. 

Although not explicitly required by the Privacy Act, agencies could record and prepare 
transcripts of advisory committee meetings and make redacted versions of them available to 
applicants. Many granting agencies, however, administer dozens of peer review committees, and are 
hard-pressed merely to prepare minutes of panel meetings. A requirement that meetings be recorded 
and transcripts prepared would probably discourage candor to some extent and would definitely add 
greatly to the expense of the peer review process. On the other hand, some agencies, like NEA, for 
reasons of their own routinely record peer review meetings. To the extent that transcripts are also 
prepared, they should be made available to applicants with information allowing the identification of 
particular speakers suitably redacted. The tape recordings themselves, however, should not be made 
available to applicants, because they would no doubt allow an applicant to attribute particular 
comments to particular individuals.*'' 



^^^emorandum to House Government Operations Subcommittee on Information, justice and Agriculture from American L.aw 
Division, Congressional Research Service, dated January 4, 1990. 

^'Memorandum to Erich Bloch, Director from Lawrence Rudolph, (undated). 

^'^ee also United States General Accounting GfTicc, Peer Review. Compliance With the Privacy Act and Federal Advisory 
Committee Act 6 (1991). 

^'^Sce text accompanying notes 501-509, infra. 
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2. Rebuttal 



As we have seen, the Privacy Act gives persons about whom an agency keeps records the right 
to demand that the agency correct any information in such records that is not accurate, relevant, 
complete or timely. Depending upon the timing of the attempted correction, this provision could 
provide an avenue for applicants to submit rebuttals to inaccurate statements in the documents that the 
agency makes available as feedback. Agency practice with respect to providing rejected applicants an 
opportunity to rebut initial peer review determinations varies widely.^ Applicants for NIH grants 
may submit purely factual rebuttals to the relevant Advisory Council, but it is highly unlikely than an 
Advisory Council will ever seriously consider a rebuttal absent flagrant factual errors. NSF's recent 
reforms do not provide any opportunity for an applicant to rebut factual errors in the review 
documents. EPA's Privacy Act regulations give applicants an opportunity to submit corrections to 
documents contained in their files, but the most important documents are usually removed before the 
applicant can gain practical access to them, and it is not clear that any submitted rebuttal would be 
considered prior to the fmal funding decision. NEA applicants can obtain a reconsideration of 
rejected applications, but only if an agency staffer is willing to take up the applicant's cause. Even in 
those rare cases in which an applicant can persuade a staffer to appeal, the grounds for 
reconsideration are very narrow. 

Of the agencies studied here, only NIH provides an opportunity for the applicant to rebut any 
errors manifested in the evaluations prior to the final funding decision, and even that process is not 
especially effective. The Privacy Act provides that a person about whom an agency keeps records 
has the right to review those record and to secure the correction of any information that is not 
accurate, relevant, complete or timely. Once again, however, this requirement applies only to 
records that the agency retains long enough to be available to a requesting applicant. Documents that 
are discarded at the end of peer review committee meetings or soon thereafter are generally not 
available to be rebutted under the Privacy Act. 

The granting agencies should follow NIH's practice of making summary statements ("pink 
sheets" in NIH parlance) available to applicants sufficiently in advance of a final decision to correct 
inaccuracies and to have those corrections considered by the agency staff. '^^ In addition, the agencies 
should retain any written evaluations of peer reviewers and minutes and any transcripts of committee 
meetings in the applicants' files for a sufficient length of time (normally less than one month) that the 
applicant can obtain a copy and rebut information contained in therein. While this may increase the 
number of document requests and attempted rebuttals from the current minimal level, it should not 
bog down the process, because only factual rebuttals (not disputes about the merits of the evaluations) 
need be considered. Finally, the existing NIH process could be improved by allowing applicants to 
submit written rebuttals to the peer reviewers themselves, rather than to a reviewing entity (like the 
NIH Advisory Councils) that usually approve the peer reviewers* recommendations on a pro forma 
basis. This would provide a realistic opportunity to affect the decisionmaking outcome without 
consuming too much reviewer time.'**^ A second meeting of the reviewers would not be required, 
except in cases in which the reviewers believed that a serious error had been made that would affect 



'^ According to Ru8trum Roy, the Dutch government allows rebutul prior to funding. Roy, gupra note 404, at 327. 

See Chubin &. Hackett. lupra note 3, at 203-04 ('Principal investigators . . . should be allowed to write a rejoinder to their reviews 
before the award . . . ."). 

A recent book on peer review by Chubin and Hackett suggesu that NSF implement a less adversarial 'dialogue* model: 

Principal investigators and authors should be allowed to write a rejoinder to their reviews before the award or publication 
decision. . . . We hope peer review can become a dialogue, a collective decision with somewhat less of the adversarial 
tone it now has. 

Chubin &. Hackett, supra note 3, at 203. 
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the outcome of the process. Any additional meeting could probably be accomplished by conference 
telephone call.'*'^ 

RECOMMENDATION 

Granting agencies should prepare and routinely make available to applicants brief summaries of the 
reasons for peer review evaluations. If peer reviewers or peer review committees prepare written 
evaluations of individual applications, these documents should be retained for a full funding cycle, 
and copies of such written evaluations should be made available to applicants upon request in a 
redact^ form so that particular evaluations may not be attributed to particular reviewers. This 
information should be made available to applicants sufficiently far in advance of the agency's final 
decision that applicants may review the documents, submit corrections of any information that is not 
accurate, relevant, complete or timely, and have those corrections consider^ by the peer reviewers 
and/or the agency staff. 

Much of the reason for the wide variation among the granting agencies with respect feedback 
and rebuttal stems from the fact that few have attempted to articulate substantive criteria and 
guidelines for implementing their Freedom of Information Act, Federal Advisory Committee Act and 
Privacy Act obligations. In some agencies (e.g. NEA) the agency's practice varies from program to 
program depending upon the particular Director's attitudes about the value of sharing information 
concerning the content of panel meetings with rejected applicants. Such wide variations across 
programs within a single agency on issues as basic as the agency's Privacy Act obligations are 
difficult to justify. On the other hand, it is not necessary that the agencies narrowly confine the 
information that they make available to rejected applicants to the bare minima required by the Privacy 
Act. 

The agencies should determine in advance and in some detail the kinds of information that will 
normally be made available to rejected applicants and to the public and adhere to those determinations 
in all of the agency's programs in the absence of compelling reasons for departing from them. The 
best vehicle for crafting agency policy on information disclosure is an informal rulemaking 
proceeding devoted to that purpose. The agency could publish a proposed rule containing a detailed 
description of the information that will normally be available to applicants and to the public and 
specifying procedural vehicles for obtaining that information. Some information (e.g. scores, 
anonymous evaluations or staff summaries of peer evaluations) could be made available automatically 
in the agency's letter notifying applicants of the outcome; other information might be made available 
only upon receipt of a specific request and only to someone authorized to receive it. The agency 
could even create an appeals process under which an applicant or member of the public could demand 
more information upon a showing of good cause. 

RECOMMENDATION 

Granting agencies should, through notice-and-comment rulemaking, write uniform procedures 
identifying information that will normally made be available to grant applicants and the public and 
specifying the procedures under which particular kinds of information will be available to different 
classes of requestors. Such procedures should specify information in as much detail as possible, and 
they should be made applicable to all granting programs within a single agency. 



^'^Since the conference call meeting would be a meeting of an Advisory Committee, ii would have to be announced in advance. Since 
it would be considering an individual application, however, the meeting could be closed to the applicant and the public. Thus, the meeting 
could be accomplished by a conference call in which only members of the peer review committee would be invited to attend. 
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A. Anonymity 

1. Applicant Anonymity 

An obvious first-level shield against favoritism and animus is a requirement that the name and 
institutional home of the applicant be kept secret from the peer reviewers.** Like "blind grading," 
this sort of anonymity makes it difficult to reward friends or damage enemies, because the evaluator 
is not supposed to know which projects are associated with which applicants. It also should eliminate 
the "halo effect" that results in unmeritorious applications receiving funds on the basis of the 
reputation of the principal investigator. Finally, blind evaluations may increase the chances that 
meritorious applications from mavericks will be ftinded. 

The most powerful objection to applicant anonymity is the fact that in many cases the best 
indication of the potential for success is the applicant's experience and past track record. Since peer 
reviewers cannot evaluate an anonymous applicant's track record, they would be limited to the 
application itself, which may depend more on the applicant's writing ability than his or her 
competence or potential for creativity. In relatively narrow fields, it is impossible as a practical 
matter to shield the identities of applicants from reviewers. People in a given field know each other's 
work, and the citations list for a grant application is likely to be filled with citations to the applicant's 
previous work.*" Finally, blind reviewing may discourage competent reviewers from participating in 
the process.*" 

Applicant anonymity is desirable when it is feasible, but it is not often feasible. Some of the 
programs in NEA are able to achieve anonymity for portions of the review (e.g., blind auditions of 
orchestras or performers via tape recordings), but they lift the veil of anonymity for other portions. 

RECOMMENDATION 

Where feasible and consistent with effective application of merit review criteria, the identities of 
applicants for discretionary grants should not be revealed to peer reviewers. Agencies should 
consider allowing peer reviewers to conduct discrete portions of peer reviews under conditions of 
applicant anonymity in cases in which complete applicant anonymity is not feasible or consistent with 
effective application of merit review criteria. 

2. Reviewer Anonymity 

Many observers of the peer review process have suggested that the identity of peer reviewers 
ought to be a matter of public record. Biased reviewers can hide behind the cloak of anonymity and 
thereby avoid accountability to applicants and the general public for their biased evaluations. 
Opening peer review to public inspection could shield the decisionmaking process against unfounded 
charges of bias.*" Some observers argue that reviewers who make career-shaping decisions about the 
scientific quality or artistic excellence of grant proposals should be willing to defend their evaluations 



**ine following analysis addresses only the extent to which government-retained peer reviewers should be aware of the identity of 
applicanu. It does not address the broader question whether the identities of applicants should be made available to the general public, a 
question that invokes the interplay between the Privacy Act and FOIA. 

^''Sce Chubin, supra note 425. 

*"l976 House Hearing Reports, supra note 3, al 35. 

*"For example, peer review panels that make funding recommendations to the California Arts Council have public observers. 
According to Mr. Elliot Klein, Administrator of the Music Panel, 'the fact that we have public observers at our meetings also furthers 
assures the level of integrity.* Chris Pastes and Herman Wong, Arts Council Doling Out Funds Today, Lx>s Angeles Times, Aug. 28, 
1987, pt. 6. p. l,col.2. 
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forthrightly to those whose lives are directly affected. *°^ Long-time observers of the peer review 
process, Chubin and Hackett maintain that: 

At bottom this suggestion [for eliminating reviewer anonymity] is motivated by two 
questions: Is it appropriate to allow comments in a review that one would not write 
directly to the author? and How many essential, accurate, critical comments about a 
manuscript of proposal cannot be expressed in language that could be directly 
communicated to the author? Criticisms that must be anonymous probably should not 
be written, and it would be a very rare criticism that could not be phrased in civil 
language. *° 

Chubin further maintains that reviewer anonymity promotes hypocrisy in professional 
relationships: 

The dichotomy between the face-to-face civility that peers are expected to maintain 
and the behind-the-scenes savagery that peer review protects creates noticeable 
strains-and sometimes outright cleavages~in a research community. Peer review 
thus disarms and factionalizes while shielding strong personalities from any timely 
cross-examination. The public and private exchanges between researchers may differ 
significantly in style and tone.** 

Most reviewers, however, strongly object to this suggestion. »* One former NSF official with 
more than eight years of experience as a program officer found this proposal to be "hopelessly naive 
and destructive."** The primary disadvantage of eliminating reviewer anonymity is the impact that it 
would have on the candor of the evaluations.*'"' In a world in which the roles of applicant and 
reviewer are frequently reversed, a reviewer may not be as candid in a negative ev^uation of a 
project. Even if all reviewers could somehow be absolutely shielded from retaliation, few members 
of a professional community are anxious to incur the hostility of a colleague by making sharply 
critical evaluations, even when warranted. As with nonanonymous faculty evaluations of students in 
job or professional school applications, the reader of the nonanonymous evaluation must have a keen 
eye for the slightest indication that the writer is subtly damning the subject with faint praise. As 



^'^Opponenu of reviewer anonymity at the 1976 House subcommittee hearings on peer review at NSF relied upon the following 
somewhat defensive arguments: 

(1) Qualified reviewers can be relied upon lo be candid and straightforward in their evaluation of a proposal, regardless of whether 
the system is open or closed. Reviewers who are not willing lo defend their positions in an open system ought not to be reviewing Federal 
grant proposals in any case. 

(2) Openness would result in more responsible and objective reviews. Superficial or personality-based comments would diminish, and 
the result would be a more focused and effective evaluation. 

(3) Qualified scientists will continue lo participate in the interest of furthering the best quality science. 

(4) Confidenlialily makes the system unnecessarily difficult to defend from charges of internal bias, old-boy practices, favoritism, or 
other criticisms. That is, an open system can better demonstrate iis impartiality and effectiveness. 

(5) An open system would increase the workload of program officers in some ways and reduce it in others. In any event, staffing 
levels can be adjusted to the new workload. 

(6) Openness means change and change may cause some problems initially; but, in the long run, the system will recalibrates and 
stronger reviews will result. The i>ew system will stand public scrutiny, and science will achieve a higher level of credibility. 

1976 House Hearing Reports, supra note 3, at 44-45. 

"'^Chubin & Hacken, supra nole 3, at 205. 

***Chubin, supra note 425. 

***C)nly 21% of ihc respondents in a survey of NCI applicants preferred eliminating reviewer anonymity, while 61% opposed it and 
12% were undecided. Chubin &. Hackett, supra note 3, at 78. 

*^ommcnu of Mr. Garth Redfield, supra note 108. 

^'^See, e.g., Voelker v. Federal Bureau of Investigation, 638 F. Supp. 571 CED. Mo. 1986)(stres8ing the need for confidentiality in 
outside evaluations for purposes of government employment); Mcrvin v. Bonfanti, 410 F.Supp. 1205 (D.D.C. 1976) (same). 
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much as we might wish it were otherwise, it is simple human nature not to speak candidly when the 
subject of the conversation is in the same room.** 

If the anonymity of peer reviewers were lifted, there could be a dramatic increase in overt and 
subtle "lobbying" of peer review members over the phone, at lunch, and in other contexts in which a 
potential applicant had an opportunity to buttonhole a reviewer. Although lobbying in the political 
context is justified as an attempt to convey information to poorly informed decisionmakers, it is 
generally viewed by the public as conduct just short of bribery. In context of discretionary grants, 
the peer reviewers have a predefined source of information (the grant application) that is meant to 
constitute the exclusive decisionmaking "record" upon which the rankings are based. No additional 
information is necessary or desirable. Therefore, most agency staffers, reviewers and applicants 
view overt lobbying as highly inappropriate in the grants process. To the extent that eliminating the 
reviewers' shield of anonymity opens up the entire process to the specter of lobbying, it is highly 
undesirable. 

All of the agencies studied in connection with this report refused to reveal to applicants the 
identities of professionals who conducted detailed reviews of individual proposals. While protecting 
the identities of primary and secondary panel reviewers, NIH does make available to applicants and 
the public the names of all of the members of the panel. Similarly, the identities of EPA panelists are 
ascertainable to applicants who go to the trouble of attending panel meetings.*" The applicant may 
then attempt to guess the identities of his or her primary and secondary reviewers among the 10-15 
members of the panel. As a practical matter, it would be difficult in any event to shield the identities 
of panel members from applicants, because "it is impossible to keep secret the names of members of 
a standing committee who travel to Washington several times annually."^'" Since a person who wants 
to ascertain the identities of panel members can probably do so with some effort, they should be 
easily available to everyone. 

As we have seen, the FOIA does not require that the identities of reviewers be revealed, because 
they are protected by the internal deliberations exemption to that Act's disclosure requirement. 
Reviewer anonymity is more difficult to protect under the FACA, because the identities of panel 
members must be revealed when the committees are chartered and when they meet in open session. 
The identities of the applications that a particular panel considers, however, need not be made public 
if the committee is free to go into "executive session" to consider individual applications. Although 
peer review conmiittees in all of the agencies studied here do in fact go into executive session to 
consider individual applications, it is only permissible if an open meeting would otherwise reveal 
information "of a personal nature where disclosure would constitute a clearly unwarranted invasion of 
personal privacy," a topic discussed above. Similarly, whether the Privacy Act allows agencies to 
shield the identities of reviewers from applicants (as opposed to the general public) depends upon 
whether the document revealing that information constitutes "investigatory material compiled solely 
for the purpose of determining suitability, eligibility, or qualifications for . . . federal contracts . . . 
," an exemption that was also discussed above. 

The question of reviewer anonymity under the Privacy Act is utterly unresolved. One side of 
the debate argues that the contracts exemption from disclosure is equally applicable to grants; the 
other side argues that grant and contracts are clearly distinguishable. The best way to resolve this 
impasse is for Congress to amend the act to provide applicant access to all staff-prepared summaries, 
initial written reviews by peer reviewers, and minutes of peer review committee meetings. The 
identities of the reviewers, however, should be shielded from view. 



**1976 House Hearing Reports, supra note 3, at 45-46. 

^'^SF also makes its lists of potential reviewers available to the public. These lists are generally so long that it is impossible to guess 
the identities of individual reviewers. 1976 House Hearing Reporu, supra note 3. at 46. 
^'0ld..at46. 
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RECOMMENDATION 

Congress should amend the Privacy Ac. lo clarify that discretionary grant applicants may have access 
to and submit proposed corrections to all staff-prepared summaries, initial written reviews by peer 
reviewers, and minutes of peer review committee meetings. The amendments should provide that 
granting agencies may refuse to disclose information that would allow an applicant to identity the 
names of the persons who conducted detailed individual reviews of his or her application. The 
identities of all members of any peer review panels should be made available to applicants and the 
public. 

If reviewer anonymity is a desirable goal, then it should be preserved in fact as well as in 
principle. Reports of breaches of confidentiality abound, even in institutions like NBA and NIH that 
take confidentiality very seriously. The only effective sanction currently available is removal of the 
offender from the committee and perhaps barring the offender from fiiture committees. Granting 
agencies should explore stronger sanctions. One NBA official suggested that panelists should be sent 
a strongly worded letter informing them of their Privacy Act obligations and of the penalties that may 
flow from breach of confidences. Another NIH staffer suggested that granting agencies be 
empowered to administer fines for breaches of confidentiality. Since granting agencies are not 
usually regulatory agencies and do not typically have the authority to invoke civil or criminal 
penalties, this solution might require statutory amendment. It would also require a procedure within 
the agency for finding facts and administering the fines. Still another strong sanction that is probably 
available under current law is to bar offending reviewers from receiving a grant from the granting 
agency for a period of time, depending on the seriousness of the breach. Because this penalty may be 
perceived as too harsh, however, agency staff may be reluctant to invoke it. Since no sanction seems 
clearly preferable to the others, agencies should determine the proper punishment for breaches of 
confidentiality on a case-by-case basis. 

Another serious problem with breaches of confidentiality is the difficulty of detecting instances 
of serious breaches. Almost everyone associated with the system at NIH believes that it happens 
sometimes, but most agree that it is not ubiquitou" Unless someone associated with the program 
serendipitously overhears a reviewer breaching a confidence, the most likely source of information 
concerning breaches of confidentiality are the persons to whom the confidential information was 
conveyed. Granting agencies could make confidentiality a two-way street by making it inappropriate 
to receive confidential information about the contents of peer review meetings as well as to give it. 
The agencies could promulgate regulations providing that grantees could lose their grants (and 
perhaps future grants) if they receive confidential information and do not report the source of that 
information to the granting agency within a specified period of time. The regulations would function 
much like an honor code in a university in which students are prohibited from cheating and are 
obligated to report any instances of cheating that they observe. 

A. Composition of Review Committees 

Several possible hedges against bias in peer review address the question of who may review 
particular applications. These largely prophylactic measures are aimed at preventing situations in 
which bias might infect the process from arising, rather than revealing bias as it happens In 
addition, they can help alleviate fears that too much decisionmaking power has been delegated from 
publicly accountable agency officials to unaccountable peer reviewers. 

1. Limited Nonrenewable Terms 

One way to prevent "old boy networks" from arising is to ensure that the membership of peer 
review committees changes relatively rapidly by providing that each member may only serve for a 
limited nonrenewable term. For example, most NBA panels are reappointed on an annual basis, and 
a panelist may only be reappointed for a maximum of three consecutive one-year terms. Forced 
turnover can help ensure that the committees do not become dominated by particular individuals with 



Peer Review in Discretionary Grantmaking 103 



narrow viewpoints. It does not, however, guarantee that the committees will not be dominated by 
persons wedded to a particular paradigm. In most agencies the staff is responsible for choosing the 
committee membership.*" If the staff "stacks the deck" with persons who know each other and who 
share similar ideas about the relevant issues, the effect on the outcome of the granting process will be 
fairly predictable. So long as there are plenty of persons with the same viewpoint available among 
the prospective panelists, rapid turnover is not a solution to staff bias. 

2. Random Selection from a Qualified Reviewer Pool 

The potential for "stacking the deck" could be reduced by requiring the staff to assemble a long 
list of qualified potential reviewers and selecting actual reviewers for particular proposals or groups 
of proposals through a random selection process. Although this might give potential applicants a 
greater degree of comfort that the process was functioning fairly, it would reduce "old boyism" only 
to the extent that the original pool was not limited to prominent old boys. Random selection of 
reviewers would prevent the staff from choosing the most knowledgeable reviewers to evaluate 
particular applications, and the process would present the additional burden of assembling and 
running a fair random selection process. ^'^ 

3. Youngster Participation 

Another solution to the "old boy network" problem and to any systematic reluctance of 
established peer reviewers to fund "youngsters" is to provide that a certain percentage of the 
membership must consist of relatively new researchers. In highly competitive programs with chronic 
funding shortages, the conservative bias of peer review operates to discriminate against relatively 
young researchers attempting to break into the field. Over time, these discouraged "youngsters" will 
either shift to other research fields or leave research altogether. In addition to spicing up committees 
with persons reflecting fresh viewpoints, increasing youngster participation would give younger 
applicants an opportunity to see the system at work from the inside and perhaps enhance their own 
chances of securing a grant in future competitions. 

4. Lay Person Participation 

Most granting agencies currently appoint only experts from a very narrowly defined pool of 
"peers" to peer review committees. Some observers of the peer review grants system have suggested 
that the definition of "peer" be broadened to include professionals from neighboring fields and even 
lay persons with no special knowledge about the subject matter. In this view, "(t]he scientific merit 
of a field can be judged better from the vantage point of the scientific fields in which it is embedded 
than from the point of view of the field itself."*" This view has also been advanced in the fine arts, 
which are generally more accessible to persons without particular training and experience.*'* 



*"One exception is EPA, where the stafT chooses the chairman of the peer review commitiee, and the chairman chooses the remaining 
members. The problem may be roughly the same if the stafT can choose chairmen who will predictably choose adherenu of a particular 
point of view. 

*'^1976 House Hearing Reports, supra note 3, at 36. 

*"Alvin M. Weinberg, Criteria for Scientific Choice, 1 Minerva 159-71 (1963), at 164-65 (Quoting Dr. John Von Neununn). See 
also Roy, supra note 439, at 24. 

*'*rhe former director of New York City's public art program suggesu that NEA panels be composed of eleven persons, eight of 
which would be chosen from among ordinary people, such as teachers, construction workers, housewives, and physicians. He argues that: 

The panel's discussion of what art should be exhibited would give artists a sense of what the community will support and the 
community a feeling for what the artisu are trying to do, eliminating conflicu such as the over the Mapplethorp exhibit. Had this been 
done with him, some of the offensive photographs would have been excluded without damage to his artistry. 

Moreover, the panels will eliminate the stranglehold on NEA grants that a small groups of artists and their supporters now have. They 
will provide c^portunities for grants to many artists throughout the United States who are now excluded from the NEA old-boy network. 
Furthennore, they will be effective in challenging the mediocrity of the art that is on the so-called 'culling edge' today. 

Donald Martin Reynolds, Fund Art for the Communities Sake, Newsday, Nov. 6, 1990, p. 45. 
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The tendency to choose peers from a very narrow pool limits the number of available reviewers 
and, consequently, .concentrates the workload on the shoulders of a relatively small number of 
experts. In addition, it exacerbates the tunnel vision problem and facilitates the formation of old-boy 
networks. It also provides the potential for research and even financial conflict of interest, because a 
narrowly drawn pool of peers will invariably include some potential competitors of most of the 
applicants. Finally, in very specialized fields, limiting panels to recognized professionals risks 
rendering the process unaccountable to the public that ultimately provides the funds. ^'^ 

One shield against these sources of bias would be to appoint one or more professionals from 
related fields and educated lay persons to all peer review committees.^'* Appointing lay persons to 
peer review panels should enhance the accountability of the funding agencies.'" Each lay member of 
a peer review committee would sit as a representative of the taxpayers who have no direct stake 
either personally or professionally in the outcome of the process. This would also help avoid the 
tunnel vision problem and perhaps even enhance the chance of funding innovative mavericks. 
Finally, a requirement that lay persons and professionals from related fields serve on panels would 
probably hinder any staff efforts to "stack the deck" with adherents to particular scientific or artistic 
points of view, because their attitudes would be less easily ascertainable in advance.'" Congress 
thought so highly of the concept of lay participation on peer review panels that it wrote into the 1990 
amendments to NEA's statute a requirement that each NEA peer review panel "include representation 
of lay individuals who are knowledgeable about the arts but who are not engaged in the arts as a 
profession and are not members of either artists' organizations or arts organizations."'" 

Other than the inconvenience to the lay person, the primary objection to including lay persons 
on peer review committees is that they will lack the expertise to evaluate proposals properly. A lay 
person without training in the relevant field, it is argued, cannot possibly understand the strengths 
and weaknesses of any given application, much less measure one against the other. '^ Some 
professionals in the arts similarly maintain that "knowledge, experience, talent, and interest make the 



For example, performance artist John Fleck candidly admits that urinating on the stage and simulating lovemakJng to hinruelf is art 
because professionals say it is. 

Why is it art? Because art professionals say it is . . . I've got a pile of reviews saying how wonderful I am. It isn't for everybody. 
Not everybody is going to understand it, but not everybody undersunds modem an. 

Paula Span and Carla Hall, Rejected!, supra note 326. 

Professor Joseph Epstein attributes the problem to the intellectual weakness of some fields of artistic (and presumably scientific) 
endeavor: *[I1f the field is weak, the grants are going to be shaky, because the peers are from those fields and they're giving grants to 
people like themselves.* Richard Bernstein, The Endowment: A Reflective Defense, New York Times, Aug. 14, 1990, Bl, col. 1. 

"former NEA Chairman, John Frohnmayer consistently advocated lay representation on NEA peer panels, but by 'lay* he 
apparently meant persons highly educated in the arts who did not obtain their livelihood from their art. See Barbara Gamarekian, Aru 
Nominee Speaks Out Against Helms Amendment, New York Times, Sept. 23, 1989, sec. 1, p. 9, col. 2. 

Rustium Roy suggests that peer review granting agencies have defined *peers* too narrowly and should expand the class of peer 
reviewers to iiKlude representatives from *a national cross-section of professioruils, i.e. it should be mandated that on the average, if 80% 
of the PhD's in materials scieiKe work in industry or government, then 80% of the reviewers of publicly supported research must come 
from the groups.* Roy, supra note 439, at 24, 25. The proposal here is to define go beyond the universe of peers, even broadly defined, 
to iiKlude educated professionals and lay persons on peer review committees. 

Ariene Goldbard, Co-director of the Baltimore-based Neighborhood Arts Programs Natioiial Organizing Committee, suggesu that: 
*It would be possible to design a system of cultural responsibility, where the decisions about what was worthy of support, by ar)d large, 
were made locally by the people who have some contact with the work as it's produced, who know something about the community in 
which the work is lo be distributed.* Can the Government Promote Creativity— Or Only Artists? New York Times, April 25, 1982, sec. 4, 
pg. 6. col. 1. 

"^This observation is more accurate in the sciences than in the arts. Sometimes lay persons have fairly well-defined views about 
artistic issues that are easily ascertained in advance. For example, it would be fairly easy for a suff concerned about outside criticism and 
congressiorul pressure lo reduce the probability of funding erotic art by appointing a lay person to the relevant panel whose views on 
erotic art were well known. Lay persotu generally have less well-defined ideas about highly esoteric scientific issues. 

'"45 U.S.C. §959(c)a)- 

'^At least two NIH staffers interviewed for this project argued thai there should be no public members on peer review committees, 
arguing that lay participants hinder the quality of the scientific debate. 



Peer Review in Discretionary Grantmaking 105 



opinions of experts more valuable than those of the lay person. When speaking of art, "the silly 
cliche '. . . but I know what I like,' is an unacceptable excuse for prejudice and ignorance. "^^' 

There are several answers to this objection. First, it is incumbent upon applicants for public 
support to write proposals clearly enough that educated lay persons can understand them. Second, 
with the help of the other panelists, an intelligent and interested lay person should become 
sufficiently educated about the critical issues that are on the cutting edge of a field of research. 
Third, it is not critical that the lay person understand all of the intricacies and nuances in order to 
gain a sense for the integrity with which the other panelists are applying the broad criteria to 
individual proposals, and this may be a sufficient role for lay person in highly technical fields. 
Fourth, lay representatives are currently being successfully used in many highly technical regulatory 
agency advisory committees, such as FDA drug review panels and EPA science advisory committees, 
despite their lack of technical expertise. Finally, just as war is too important to be left to the 
generals, these decisions are too important to be left entirely to highly educated professionals who 
may view the world from very narrow perspectives.^^ Perhaps the most important lesson of NEA's 
public policy fiascoes of the early -19905 is that public support can quickly vanish when ftinding 
decisions are left to a small group people who define a field of scientific or artistic endeavor, 
articulate the criteria for excellence in that field, and deem themselves the exclusive arbiters of 
excellence in that field. '^3 

Expanding the participation of lay persons or of scientists from unrelated fields will be 
successful in reducing bias only to the extent that the "outsiders" play an active role in the 
deliberations. In large programs administering many grants, the panels must of necessity divide up 
the considerable burden of reviewing individual proposals. Typically, the task of detaileil review is 
assigned to a subcommittee of one or two members of the committee who are especially familiar with 
the field of research addressed by the proposal. As a practical matter, other members of the panels 
are not likely to read the proposals in sufficient detail to offer cogent challenges to the evaluations of 
the original reviewers. Indeed, one frequent participant in peer review panel meetings in NIH 
observed that the agency staff puts pressure on panel members to accept the evaluations of the 
subconmiittees by requiring that any panel members who questions those evaluations must write up a 
justification for his or her objections. One easily implementable solution to this dilution of the voting 



^^'Salzer, supra note 393 (Ms. Salzer is a Professor of Drama at San Diego Suie University). 

Professor Karen Mulcahy testified to the independent Comnussion on the National Endowment for the Arts that: 
The decision-making of a grant panel is not the same as that of a jury at a show at an an gallery. Artistic excellence should be 
important but not controlling. It's like saying war is too imporuni a public policy to leave to the generals. To say public culture should be 
decided on by artists is like saying Pentagon policy should be turned over to defense contractors. 
Paracini, supra note 347. 

deferring to the °NEA Four,' Professor Joseph Epstein, a member of the National Council on the Arts, explained that: 

[T]he people who gave those grants are people from the field of experimenul theater and performance art who appreciate 
that sort of thing. That's part of the problem. In those fields where there are no real sundards, except a vague feeling 
that, 'gee, it works,* or, 'it clicks* there is nothing very concrete to go on. When you read the panel evaluations on 
these grants, you feel that they are intellectually very thin. 

Bernstein, supra note 515. Refering to complaints about the provision in the 1990 amendments that required lay participation on NEA 
peer review panels, an editorial in the Chicago Tribune asked: 

'Could it be that the arts community is upset because it senses that it has lost the franchise on 'aesthetic authority'? For 
twenty-five years, the NEA's 'peer-review' system allowed the arts community to enclose an increasing ingrown notion 
of aesthetic authority.' Not incidentally, it also has been able to diwy up the government kitty among its favorites, oflen 
in disregard of elementary principles of conflict-of-interest, public taste and, some cynics would say, artistic merit.' 

Editorial, Chicago Tribune, Dec. 4, 1990, p. 22. See also Margaret Spillane, The Culture of Narcissism: Performance Artist Karen 
Finley and the National Endowment for the Arts, The Nation, Vol. 251, no. 20, pg. 737, Dec. 10, 1990 (complaining that the artists' 
'specialized class-bound notion of Those Who Know What Art Is-and those who get to make and judge that art-persistently undermines 
confidence in the popular creative spirit*); Knight and Slye, supra note 386. 
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strength of the outsiders is to require that every application be read in detail by both a scientist 
familiar with the field and a person who is not an expert in that particular scientific field. 

5. Size of Committees 

One way to shield against animus and against the sort of research conflict of interest in which a 
panel member attempts to downgrade proposals of competitors is to expand the size of the 
committees. The more that a panelist's "blackball" vote is diluted by other votes that are purely on 
the merits, the less effect that vote will have on the overall rankings. Thus, committees in NIH with 
more than 25 members and committees in EPA with up to 60 members experience very little 
blackballing. Indeed, short of a conspiracy, blackballing is virtually impossible in such large 
committees. 

Large conmiittees, however, are very expensive to assemble and maintain. Travel costs alone 
can be quite substantial for a very large committee, especially if it is required to meet as many as 
three times per year. Large committees can also be difficult to manage, and they are generally less 
efficient. Uniess personal animus on the part of panel members is a particularly pressing issue (e.g., 
a field dominated by two warring camps), the added expense of expanding the size of the committee 
may not be justified by the possibility that one panelist will deliberately downgrade a proposal so 
severely that it will not be funded. In highly competitive programs in which only a few very large 
grants are awarded, however, the additional expense of assembling large committees may be 
warranted. 

6. Number of Committees 

Another way to shield the peer review process from animus and favoritism is to conduct a multi- 
tier review in which more than a single committee has responsibility for evaluating each proposal. 
Many peer review granting agencies have a high level advisory committee made up of very prominent 
people who look over the results of the original peer review committees and have the power to 
change the results (or recommend that the agency change the results). The Advisory Councils at NIH 
and the National Advisory Council on the Arts are two examples. This second tier review, hcwever, 
is usually quite superficial and rarely delves into the merits of individual proposals. Although they 
have many important functions, high level review panels do not have the capacity to ferret out and 
eliminate bias. 

The multi-tier process in NSF's Science and Technology Research Centers Program offers a 
good model for programs that make a few large grants. In that process, a proposal is initially 
distributed to outside mail reviewers and then to a six-member peer review committee for the purpose 
of culling the applications down to a manageable group of thirty applications. Additional experts are 
assembled into site visit teams that visit the thirty sites and report back to still another "external peer 
review committee" that examines all of the information available on the thirty institutions in two 
separate groups. After each group reevaluates the results of the other group, the full external peer 
review committee identifies the fifteen or so applications that it recommends for funding. 

While this extremely thorough system effectively shields the decisionmaking process from 
animus and favoritism, it is exceedingly resource-intensive, and it would not generally be justified for 
programs that award a large number of small grants. The very last step, however, is relatively 
inexpensive, and might be useful even for very large programs. The prospect of having a second 
subcommittee examine the results of the first subcommittee's evaluations should caution panelists 
against attempting to blackball proposals by giving them very low scores. 

RECOMMENDATION 

Granting agencies should not limit the pool of available peers to narrow professional fields, but 
should instead attempt to appoint reviewers from related professional fields to peer review panels. 
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Granting agencies should appoint at least one lay person without training in the relevant field to every 
peer review panel. Granting agencies should also attempt to include younger professionals who have 
not previously received grants. Granting agencies should provide that the membership of peer review 
panels changes on a regular basis. To the extent consistent with agency resources and depending on 
the size and number of the grants awarded in a program, granting agencies should consider 
maximizing the number of persons serving on an individual peer review committee, performing 
duplicate reviews in two or more subcommittees, or appointing multiple committees to perform a 
tiered review. 

7. Conflict of Interest 

The granting agencies are by-and-large sensitive to the problem of financial conflict of interest,^ 
and they generally prohibit a reviewer from participating in the evaluation of his or her own proposal 
or a proposal from his or her own institution. '^s NIH and NSF regulations allow a person to sit on a 
panel that reviews an application from that person's own institution,'^ but require that person to be 
absent from the discussions of the institution's application. A virtually identical provision in NEA's 
regulations was severely criticized on the ground that the conflict still existed even though the 
affected panelist was absent.'^ As was the case with NEA, there should be a sufficient number of 
qualified reviewers available to evaluate a panel's applications without drawing on persons from 
institutions with applications pending before that panel. '^ 

The agencies have generally failed to pay sufficient attention to the problem of research conflict 
of interest. The conflict of interest regulations of only one of the agencies studied in this report 
addressed the possibility that a reviewer could use his or her position to decrease the probability of 
success of applications from competing scientists.'^ And only one agency dealt with the possibility 
that reviewers might use information gleaned from the peer review process to their own financial or 
research advantage. Some agency officials opined that, like plagiarism, research conflict of interest 
would be classified as scientific misconduct and therefore subject to sanctions such as removal from 
peer review committees or debarment from the grant process. 

The research conflict of interest problem can best be addressed through amendments to agency 
conflict of interest regulations. The most effective solution is a complete bar on a reviewer's 
participation in the consideration of his own application or one from a close associate or the 
institution that employs him and to prevent him from evaluating any application of a rival or rival 



'^^The prototype recusal requirements are those of NIH, which exclude from study sections applicants, fanuly of applicanu and their 
co-workers both at their institutions and at other institutions. In addition, a panelist must leave the meeting when the panel is considering 
an application from his or her own otf anization or an organization with which he is negotiating for future employment. 

'^'Xhe existing conflict of interest guidelines only obliquely address applicants who are former students of panelists, a phenomenon 
that occur with some frequency in the sciences, though perhaps with less frequency in the arts. EPA has a rough rule of thumb that a 
panelist need not recuse himself from the consideration of proposals from former undergraduate students or from former graduate students 
who have not graduated within the last five years. This seems like a reasonable approach to reducing the appearance of favoritism toward 
proteges. The problem is not sufficiently widespread or disturbing, however, thai the rule of thumb should be written into a binding 
regulation. Peer reviewer should be asked at the outset of the process to identify any former students among the applicants, and they 
should be asked to recuse themselves in accordance with the five-year rule of thumb. 

^ince NSF reviewers meet as panels much less frequently than NIH reviewers, this issue does not arise in NSF as often as it does in 
NIH. The problem does arise, however, in the Science and Technology Centers Program and is dealt with by dividing the large panel of 
thirty or so panelisu into two groups and ensuring that no panelist sit in the group that performs the first review of the applications from 
that panelist's institution. See Part 3. 

'"See Part 5. at notes 84-84. 

^^This may require that some attention be paid to the definition of "same institution.* For example, if the University of California, 
Penn State University, or the State Universities of New York could conceivably be viewed as a single institution this recommendation 
could be troublesome. A better starting point for a definition of 'institution' would be a single campus of a single university. 

'^^IH's conflict of interest regulations suggest that a reviewer 'should not participate in the review of an application from a scientist 
with whom the reviewer has had long-standing differences which could reasonably be viewed as affecting objectivity.' NIH Manual 4510, 
supra note 51, at 7; NIH Manual 1805 at 6-7. 
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institution in a competition for a pool of funds from which he or his institution is competing. 
Considerable criticism in the press lead Congress to amend NEA's statute to adopt this solution, and 
it does not appear to have hampered the peer review process at that agency unduly. 

This prophylactic solution is probably feasible for many scientific research funding programs. If 
scientists conducting unrelated research are qualified to evaluate proposals in the particular field and 
if they are willing to devote their time to evaluating research in fields not directly related to their own 
research interests, then a ban may be appropriate. In some programs^ however, this constraint may 
unduly limit the supply of qualified and willing reviewers. For example, in NSF's Science and 
Technology Centers Program, where virtually every major research institution in the country either 
has or is applying for a grant, it is difficult to find qualified scientists for mail reviews who are not 
associated with a competing institution. Thus, it may not always be feasible to bar persons with 
potential research conflicts of interest from peer review committees, even though these are exactly the 
persons who can nriost effectively blackball rivals and expropriate novel ideas contained in research 
proposals. 

Alternatively, all members of peer review committees could be required to sign an oath that they 
will not use any information gleaned from the review process to further their own research interests. 
To some extent, the other members of the panel can police against research conflict of interest, 
because they may be familiar with each others' research and therefore know whether one of their 
number may inappropriately gain access to information about research in a competing laboratory. 
The problem with this solution is both the difficulty of proving that a researcher violated the oath and 
the chilling effect that the oath might have on reviewers. It may be that few researchers would be 
willing in advance to subject themselves to sanctions if a rejected scientist later accused them of 
appropriating research ideas. 

Still another suggestion might be to promulgate a general rule prohibiting reviewers from using 
information and ideas gained in peer review committees in their own research absent the written 
consent of the applicant. Any applicant who believed that his ideas were appropriated could lodge a 
complaint with the funding agency. If, after a hearing, the claim were proved to be valid, the agency 
could suspend further funding of the offender's research or condition suspension on some more 
appropriate penalty, such as a public retraction and apology. 

In the final analysis, the remedy for research conflict of interest is the professional integrity of 
the researchers who sit on the committees and the willingness of other scientists to look unfavorably 
upon the abuse of the peer review process to appropriate the ideas of others. The Director of an 
important NSF program observed: 

As to conflicts of research interests, this is something you really can't legislate. If 
you try to define it and write it down, people will use it as a crutch. Just like 
Keating in the S & L scandal - if you write down a rule, then people use it as a 
reason to get by and play games, saying you said this, but you didn't say this. They 
will play games with the interpretation of the rule. You just can't legislate 
morality.'* 

Yet there is currently no explicit prohibition of the practice and no remedy for the damage done 
by this abuse of the peer review process in the conflict of interest regulations of any of the agencies 
studied in this report. Indeed, it is even possible that some scientists view this access to the ideas of 
others as appropriate compensation for the time spent reviewing proposals."' 



''telephone interview wilh Dr. Bill Harris, supra note 431 . 
'^'See Chubin & Hacken, supra ncHe 3, al 205. 
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RECOMMENDATION 



Granting agencies should promulgate conflict of interest regulations that prevent any person from 
reviewing or sitting on a panel that reviews his or her own grant application or the application of a 
close collaborator, a recently graduated former student, or an affiliated institution. The regulations 
should provide that when the agency asks a reviewer to review the application of a close competitor 
or competing institution, that fact must be disclosed to the agency, to other members of any peer 
review committee and to the applicant before the time to ask for reconsideration or appeal has 
expired. To the extent feasible, agencies should avoid asking a reviewer to evaluate the application 
of a close competitor or competing institution. If necessary, granting agencies may provide for 
specific waivers of the conflict of interest recommendations on a case-by-case basis where there is no 
other practical means for securing appropriate expert advice on a particular grant application. 

8. Peremptory Challenges 

One fairly inexpensive way to enhance the perception of fairness in the peer review process is to 
allow an applicant to nominate one or more researchers for exclusion from the list of peers available 
to evaluate his or her work. An applicant could be invited to submit a short list of persons that the 
applicant believes tobe biased with respect to his or her research. The applicant might be required 
to provide reasons for exclusion or to choose from a list of reasons deemed appropriate by the 
agency. Appropriate reasons could, for example, include animus or financial or research conflict of 
interest. Lack of qualifications, however, should generally not be a sufficient reason for a 
peremptory challenge. The agency staff is in the best position to determine who is qualified to 
review an application. 

Providing an applicant one or two peremptory challenges should help reassure rejected 
applicants that they were not blackballed for personal or idiosyncratic reasons. It may also increase 
the comfort level of applicants (especially in programs in which for-profit institutions are competing 
for ftinds) that their ideas will not be expropriated. We have seen that the staffs of NIH and NSF 
occasionally allow peremptory challenges on an informal basis, and this appears to add to the general 
perception that the programs are being run fairly. "^ 

The primary disadvantage with peremptory challenges is that they deplete the supply of potential 
reviewers. In very narrow fields, the only qualified reviewers may be potential competitors or 
existing rivals. Many smaller programs have only a single peer review panel that is competent to 
review some proposals. In those programs, the peremptory challenge would result in the removal of 
the challenged individual from the deliberations. The potential contribution that the excluded 
individual may have on the panel's deliberations on other applications may outweigh the potential for 
bias with respect to the applicant making the peremptory challenge. For these reasons, the number of 
peremptory challenges allowed to any individual applicant must be limited. 

Another objection to peremptory challenges is that in some fields, it may allow applicants "to 
influence the review of their proposals by. challenging the most competent and incisive reviewers.""' 
Although limiting the number of challenges would not eliminate such attempts to "game the system," 
the agency could discourage such strategies by requiring challengers to provide adequate reasons for 
their challenges as suggested above. 



"^One question that might arise if peremptory challenges are allowed is whether potential peer reviewers could obtain such challenges 
under FOiA. The short answer to that question would seem to be that any written challenge would be available under FOIA, except to the 
extent that information contained therein feel within one of the Act's exemptions. Although this is not the place for an extended 
examination of this issue, such challenges would normally not contain trade secret information. Although clearly related to future 
deliberations, the request itself would probably not come within the exemption for internal agency deliberations. Nor is it clear that 
disclosure of the information would constitute a "clearly unwarranted invasion of personal privacy.* The person making a peremptory 
challenge has no obvious privacy interest in shielding that fact from the object of the challenge. 

"'Commenu of Mr. Robert A. Papetli, supra note 204. 
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RECOMMENDATION 

Granting agencies siiould by regulation provide that any applicant may submit a confidential list 
containing a small number of potential reviewers that the applicant deems objectionable together with 
a statement of the reasons for the challenges. The agencies should presumptively grant such 
challenges, unless the agency determines that a qualified group of peers cannot be assembled if all 
such challenges are honored. The agencies should provide, on a confidential basis, reasons for 
rejecting such challenges to the applicants who submit them. 

9. Eliminating High and Low Scores 

Most goveniment funded programs involving discretionary grants have enough funds available to 
fund fewer than one-half of the applications submitted to them. In programs in which there is intense 
competition for limited funds, a single peer reviewer can affect the outcome at the margins by giving 
an application a very high or very low score. Panels with large numbers of members can borrow 
from the practice of athletic events involving qualitative evaluations (e.g. figure skating and diving) 
and eliminate the high and low scores from the calculations of the average score. Panels with more 
time to spend on discussing the merits of applications can have the agency staff identify "outliers" 
and request that the Panel devote further attention to those applications in which one or more 
individual scores varied from the mean by more than a predetermined amount. 

RECOMMENDATION 

Peer review committees that rely upon qualitative scoring systems for evaluating and ranking 
discretionary grant proposals should develop vehicles for either eliminating or further discussing 
individual scores that vary widely from the average. 

G. Lobbying Peer Reviewers and Staff 

One of the most significant advantages of the peer review model is that it discourages lobbying 
and influence-peddling. The professionals and lay persons who participate in the peer review process 
understand that decisions should be based upon the information contained in the applications, and not 
upon information and arguments supplied ex pane by interested outsiders. Tlie peer reviewed 
granting decision is supposed to be made objectively on the basis of the application and the 
reviewers' assessment of the qualifications of the applicant. With respect to ex pane contacts, the 
adjudicatory model seems entirely appropriate in the granting agencies that rely upon peer review. 

Lobbying upper level advisory committees and upper level agency staff is an inappropriate 
technique for securing discretionary grants and therefore should be prohibited. At the very least all 
attempts to sway peer reviewers and agency staff outside the formal confines of proposal submittal 
and review should be noted in writing by the object of the importuning. Contact between lower level 
agency staff (e.g. Program Directors in NIH) and applicants should be encouraged, but direct 
importuning of peer reviewers and upper level staff with the power to accept or reject the 
recommendations of peer reviewers should not be allowed. 

RECOMMENDATION 

Granting agencies should encourage informal contacts between applicants and agency staff who do 
not participate in the decisionmaking process for the purpose of conveying information and providing 
advice. Agencies should adopt regulations subjecting ex pane contacts with peer reviewers and with 
agency staff in a decisionmaking capacity to the strictures of sections 554 and 557 of the 
Administrative Procedure Act. 
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H. Levels of Review 



Several NIH officials interviewed for this report maintained that the Advisory Councils offered 
so little substantive input into the ultimate funding decisions that the Councils should be abolished. 
The primary complaint about Advisory Councils was that they did not offer any real scientific input. 
To some extent this was attributed to the presence of lay members on the Advisory Councils. One 
NCI staffer suggested that the Advisory Councils be replaced by the Boards of Scientific Advisors 
that already exist in each of the Divisions. In NSF lay persons generally do not sit on the peer 
review panels that are sometimes assembled to aid the agency staff, but the overall policymaking 
entity, the National Science Board, theoretically has the power to make the final determinations. 
This power is exercised frequently in the context of large programs like the Science and Technology 
Centers Program, but it is almost never used in the context of programs that make a large number of 
relatively small grants to individual researchers. 

A very persuasive case can be made, however, for the proposition that funding decisions are too 
important to be left entirely to the scientists in the field in which the research is funded.''^ The 
funds, after all, come from the public purse, and the public can reasonably insist that funding 
decisions reflect criteria other than the scientists' view of scientific merit or the artists' view of 
artistic merit. Indeed, if important funding decisions were left entirely to persons who were not 
government officials, the process might raise constitutional concerns under the delegation doctrine.^" 
In both the arts and the sciences, priority-setting invariably involves policy considerations that are 
intensely political in nature. While the agency staff is capable of incorporating externally articulated 
policy considerations at the final stage in which the ultimate funding decisions are made, the staff is 
also composed of highly trained professionals. In addition, lay participants can occasionally lend a 
perspective that enhances the quality of the scientific review. ^^^ An institutional vehicle incorporating 
lay members may be a necessary quid pro quo for continued funding of science on a large scale by 
government. 

I. Rotating Agency Staff 

In some of the agencies studied in connection with this Report, the agency staffer responsible for 
a grant has a great deal of influence on the ultimate success or failure of the application, while in 
others his or her influence is marginal. For example, in NIH, the Scientific Review Administrator 
is supposed to be a silent observer of peer review meetings, speaking only when spoken to, and he or 
she plays only a limited role at Advisory Council meetings and in the upper level decisionmaking 
process. The NSF Program Officer, on the other hand, is responsible for ranking of the proposals 
and making the initial recommendations whether or not to fund, and the outside mail reviewers are in 
a sense merely advisors to the Program Officer. One former NSF peer reviewer, who is now a 
staffer responsible for grants in another agency, reported that it is not uncommon for the NSF 
Program Officer to suggest to the outside reviewers subtly how they might view the proposals. 
Many interviewees noted particularly the powerful position occupied by the NSF Program Officers. 
Agency staffers with biases of their own can distort the peer review process so that it does not result 
in funding the most scientifically meritorious proposals. 

One relatively expensive solution to the problem of staff bias is to eliminate mail reviews or 
supplement them with panel meetings in all cases. For agencies with large numbers of applications 
and relatively low travel budgets, this may be impractical. A considerable advantage of mail review 



^^ee text accompanying notes 513-523, supra. 

"^e Carter v. Carter Coal Co.. 298 U.S. 238 (1936). 

^^^or example, the principal investigator in an ongoing NCI study involving women and breast cancer proposed to change the diet of 
2,000 women for 10 year. The proposal raised large policy issues appropriate for consideration by lay persons. The Advisory Council, 
however, recommended that the proposal be rejected on the largely scientific ground that the intense publicity about the link between 
cancer and diet would probably result in dietary changes in the control population that could render the results invalid. Telephone 
Interview with Leo Buscher, Grants Management Officer, National Cancer Institute. Thursday. Oct 4. 1990. This evaluation no doubt 
benefited from the lay perspective represented on the Advisory Council. 
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is the difference in price between a postage stamp and an airplane ticket. As valuable as unbiased 
peer review is to the process of awarding grants, monies spent on the process itself cannot be used on 
the research that the program is meant to support. 

A much more easily implemented device for shielding the process from staff bias is to rotate the 
staff periodically to ensure that no single staffer establishes his or her own "barony" and thereby 
influences the course that research takes in an entire field. This solution has the disadvantage of 
depriving the system of the expertise that the staffer might acquire over the years in a single 
program, but that disadvantage might easily be outweighed by the advantage of interjeaing fresh 
blood into the system. It would also have the indirect effect of enhancing the influence of the outside 
reviewers on the ultimate funding outcome. 

RECOMMENDATION 

Agencies that administer multiple programs for awarding discretionary grants should consider 
rotating staff periodically among the programs. Especially in programs in which peer reviewers do 
not meet to reach collective judgments, agencies should rotate the staff responsible for making initial 
funding recommendations. 

J. Greater Role for Agency Staff 

Perhaps the best shield against animus and favoritism is vigilance on the part of the agency staff 
serving the peer review committees. Staffers who attend peer review panel meetings can see the 
expressions on the faces of the reviewers as they make what may be sarcastic or ad hominum 
comments. The staff can also tell when a panelist has crossed the fme line between advocacy on the 
merits and favoritism for an "old boy" or hallowed researcher. Agency staffers are also generally 
more inclined to view mavericks with a sympathetic eye. If the staffer has the de facto power to 
discount votes that he or she believes are biased, the objectivity of the system is enhanced. 

On the other hand, it may be expecting too much of a busy staffer to draw such fine lines. Most 
staffers have training in the subject matter that the panel addresses, not in psychology or politics. 
Empowering the staff to reverse or discount panelist votes may simply invite the staffer to second 
guess the experts on the merits or to express his or her own biases. Once the agency has decided to 
draft rules or guidelines on bias and conflict of interest, however, the staff can play a very important 
role in ensuring that peer reviewers are well-educated about them and that they strictly observe them. 

K. The Audit Possibility 

The openness in the peer review grants process inspired by the Privacy Act, the Freedom of 
Information Act and the Federal Advisory Committee Act should go a great distance toward 
insulating the process from animus and conflict of interest. Because it relies upon the policing 
activities of rejected applicants, however, the openness solution to the bias problem may lack efficacy 
in the context of a chronically underfunded program in which unsuccessful applicants who rock the 
boat are easily identified and shunned. In addition, openness by itself does not effectively address 
favoritism. Even in an open system, the victims of cronyism or the old boys' network are not likely 
to know that less meritorious proposals have been elevated over theirs for irrelevant reasons. Most 
rejected applicants will at most focus on the comments and review summaries that the agencies 
provide to them about their own applications. They are not likely to examine the entire proceedings 
to see if the process has been systematically or individually biased toward a particular person or 
methodological approach. 

One well-known technique for spotting decisionmaking that is tainted with inappropriate or 
irrelevant considerations is the audit, under which outside auditors selectively examine individual 
decisions in considerable detail for evidence of bias or other impropriety. The audit idea surfaced 
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during congressional hearings into alleged bias at NSF conducted during the mid-1970s."' Although 
an extensive report by the subcommittee recommended that NSF adopt an internal auditing program, 
little came of the recommendation for many years. ^'^ In 1989, NSF implemented a version of the 
audit idea with its "Committee of Visitors" program under which each Directorate must appoint a 
committee of outsiders every three years to review the peer review process in operation."* 

It should be feasible to import the audit concept to scientific decisionmaking in granting agencies 
that use the peer review model. Agencies like NSF and NIH could either appoint or hire outside 
auditors to examine randomly selected files, including applications, written comments, telephone 
logs, conflict of interest statements, committee meeting audiotapes or transcripts. The auditors would 
probe the relationships between NSF staff and reviewers and between reviewers and applicants for 
financial and research conflict of interest. The audit would be strictly confidential; the names of 
reviewers and the contents of the reviews would not be made public. If the examination revealed 
evidence of bias or impropriety, the auditors could be empowered to interview participants and to 
write a report stating their conclusions about whether the process operated fairly and in accordance 
with agency regulations. In those, presumably quite rare, cases in which bias was detected, the 
agency could decide to investigate the matter more vigorously. 

The audit idea has several advantages. First, it allows a motivated person or group of persons to 
conduct an intense review of a limited number of funding decisions without jeopardizing the critical 
confidentiality of the process. At the same time, the ever-present prospect of an audit should act as a 
substantial disincentive for reviewers and staff to bend the rules or to invoke improper 
considerations. If they were conducted by outsiders, the audits could also shield the agencies from 
charges of whitewashing by congresspersons and the general public. Over time, audits should 
provide useful evidence of the extent to which bias exists at various granting agencies.** If little 
evidence of bias resulted from a large number of audits, they could be eliminated. 

Audits have disadvantages as well. They will require resources that might otherwise be devoted 
to the pursuit of the agency's primary goals. A limited system of audits, however, should not be too 
expensive. The prospect of being audited might discourage qualified reviewers from voluntarily 
participating in the process. And it is always possible that an overly zealous auditor will erroneously 
find bias in a process in which it did not exist, thereby disrupting the process for a time. 

Despite its expense, it may be possible to implement the audit idea on an experimental basis. If 
it proved too expensive, it could be abandoned with little loss in time or energy. If it proved 
successful, it could inspire renewed confidence in the peer review process. 

RECOMMENDATION 

Granting agencies that rely upon peer review should experiment with random audits of the peer 
review process for bias and conflict of interest. 

L. Conclusion 

Despite its flaws, the peer review system is still the best model for making complex and 
exceedingly difficult decisions about how to allocate limited collective resources to the arts and 
sciences. One need only examine the output of NIH, NSF, EPA and NEA to conclude that the peer 

"'l976 House Hearing Reports, supra noie 3. 

^^The oubcommillee recommended that NSF conduct random audits of the decision processes in NSF for individual grants: 
These audits should be performed in sufficient detail to address questions of relations among Foundation staff members, applicanu, and 
reviewers. Membera of the auditing staff should not normally have responsibilities in the process of decision-nuking for individual 
awards. The audits will not replace any activities of the General Accounting Office or other Congressional arms. 
1976 House Hearing Reports, supra note 3, at 58. 

^^'See discussion in Part 3 . 

**1976 House Hearing Reports, supra note 3. 
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review model has produced marvelous results. The changes advocated here do not go to the heart of 
the peer review system. If they are not implemented, the peer review system will continue to 
function reasonably well. Yet there is little reasons to remain satisfied with any flaws if they can be 
reduced or eliminated at little cost. With some modest improvements, the model that has evolved in 
this country should provide an example to the world of how a democracy can employ expertise in the 
service of artistic and scientific excellence. 



